| # Parameter Framework (PF) test plan |
| |
| ## Notes |
| - The unit test code MAY be able to list implemented scenarios |
| in this same BDD format in order to check scenario coverage. |
| - All scenarios implicitly start by _"GIVEN a parameter framework"_. |
| - If a scenario does not specify a config files, |
| the following minimal files SHOULD be used: |
| |
| Top level file (replace `{structurePath}` by its value): |
| ~~~ xml |
| <?xml version='1.0' encoding='UTF-8'?> |
| <ParameterFrameworkConfiguration SystemClassName='test' |
| TuningAllowed='false' ServerPort='1'> |
| <SubsystemPlugins/> |
| <StructureDescriptionFileLocation Path='{structurePath}'/> |
| </ParameterFrameworkConfiguration> |
| ~~~ |
| |
| Structure file: |
| ~~~ xml |
| <?xml version='1.0' encoding='UTF-8'?> |
| <SystemClass Name='test'> |
| <Subsystem Name='test' Type='Virtual'> |
| <ComponentLibrary/> |
| <InstanceDefinition> |
| <BooleanParameter Name="placeholder"/> |
| </InstanceDefinition> |
| </Subsystem> |
| </SystemClass> |
| ~~~ |
| |
| ## Starting the Parameter Framework |
| |
| - set a logger |
| - [X] Scenario: Default logger |
| * WHEN no logger is set |
| * THEN start should succeed |
| |
| - [X] Scenario: No logger |
| * WHEN a nullptr logger is set |
| * THEN start should succeed |
| |
| - [X] Scenario: Logger should receive info and warnings |
| * GIVEN config files that emit warnings |
| * GIVEN a logger that store logs |
| * WHEN the record logger is set |
| * THEN start should succeed |
| * AND_THEN the logger should have stored info and warning log |
| |
| - [X] Scenario: Unset logger |
| * GIVEN config files that emit warnings |
| * GIVEN a logger that store logs |
| * WHEN the record logger is set |
| * AND_WHEN a nullptr logger is set |
| * THEN start should succeed |
| * AND_THEN the record logger should NOT have stored any info or warning log |
| |
| - set/get various properties before startup: |
| - "force no remote interface" |
| - [ ] Scenario: Tuning OK |
| * GIVEN config files with tuning enabled on a valid port |
| * WHEN the remote interface is NOT forbiden |
| * THEN start should succeed |
| |
| - [ ] Scenario: Invalid tunning |
| * GIVEN config files with tuning enabled on an invalid port (0?) |
| * WHEN the remote interface is NOT forbiden |
| * THEN start should fail |
| |
| - [ ] Scenario: Forbiden invalid tunning |
| * GIVEN config files with tuning enabled on an invalid port (0?) |
| * WHEN the remote interface is forbiden |
| * THEN start should succeed |
| |
| - "failure on missing subsystem" |
| - [ ] Scenario: Plugin OK |
| * GIVEN config files with a valid plugin |
| * WHEN the missing subsystem policy is set to failure |
| * THEN start should succeed |
| |
| - [X] Scenario: Wrong plugin |
| * GIVEN config files with a invalid plugin |
| * WHEN the missing subsystem policy is set to failure |
| * THEN start should fail |
| |
| - [ ] Scenario: Wrong plugin but fallback |
| * GIVEN config files with a invalid plugin |
| * WHEN the missing subsystem policy is set to success |
| * THEN start should succeed |
| |
| - "failure on failed settings load" |
| - [X] Scenario: Settings OK |
| * GIVEN config files with a valid settings file |
| * WHEN settings load failure is requested to abort start |
| * THEN start should succeed |
| |
| - [X] Scenario: Wrong settings |
| * GIVEN config files with a invalid (non existant?) settings file |
| * WHEN settings load failure is requested to abort start |
| * THEN start should fail |
| |
| - [X] Scenario: Wrong settings but ignore |
| * GIVEN config files with a invalid (non existant?) settings file |
| * WHEN settings load failure is requested to abort start |
| * THEN start should succeed |
| |
| - "schema folder location" and "validate schema on start" |
| - [ ] Scenario: Schema OK |
| * GIVEN config files with correct default schema location |
| * WHEN schema folder location is left to default |
| * WHEN schema validation is enabled |
| * THEN start should succeed |
| |
| - [ ] Scenario: Inexisting schemas |
| * GIVEN config files with correct default schema location |
| * WHEN schema folder location is set to an invalid location (/doesNotExist ?) |
| * WHEN schema validation is enabled |
| * THEN start should fail |
| |
| - [ ] Scenario: Inexisting ignored schemas |
| * GIVEN config files with correct default schema location |
| * WHEN schema folder location is set to an invalid location (/doesNotExist ?) |
| * WHEN schema validation is disabled |
| * THEN start should succeed |
| |
| - [ ] Scenario: Inexisting used schemas |
| * GIVEN config files with custom schema location (./shemaCustomFolderName) |
| * WHEN schema folder location is set to the custom shema location |
| * WHEN schema validation is enabled |
| * THEN start should succeed |
| |
| - [ ] Scenario: Schema OK, config KO |
| * GIVEN incorect config files with correct default schema location |
| * WHEN schema validation is enabled |
| * THEN start should fail |
| |
| ### Error cases/special cases |
| |
| - invalid configuration or a file can't be read: |
| - [X] Scenario: invalid top level config file |
| * GIVEN an incorect (empty ? wrong balisa ?) config top level file. |
| * THEN start should fail |
| |
| - [X] Scenario: invalid structure |
| * GIVEN an incorect (empty ? wrong balisa ?) structure file. |
| * THEN start should fail |
| |
| - [X] Scenario: invalid settings |
| * GIVEN an incorect (empty ? wrong balisa ?) settings file. |
| * THEN start should fail |
| |
| - plugins can't be found (unless silenced by the user) |
| - Done in "failure on missing subsystem" |
| |
| - usage of `<xi:include .../>` |
| - [ ] Scenario: Arbitrary xincluded node |
| * GIVEN a top level file xincluding a file with the "SettingsConfiguration" balisa |
| * THEN start should succeed |
| |
| - plugins: |
| - non-empty "Location" attribute |
| - [ ] Scenario: Plugin location OK |
| * GIVEN a top level file with a plugin folder location attribute |
| * THEN start should succeed |
| - empty "Location" attribute |
| - relative path |
| - [ ] Scenario: Empty Location, standard plugin install |
| * GIVEN config files with an empty location attribute |
| * GIVEN a plugin installed in a dlopen standard path |
| * GIVEN that "Location/Plugin[name]" is the plugin name |
| * THEN start should succeed |
| |
| - [ ] Scenario: Empty Location, non standard plugin install |
| * GIVEN config files with an empty location attribute |
| * GIVEN a plugin NOT installed in a dlopen standard path |
| * GIVEN that "Location/Plugin[name]" is the plugin name |
| * THEN start should fail |
| |
| - absolute path |
| - [ ] Scenario: Empty Location, plugin absolute path |
| * GIVEN config files with an empty location attribute |
| * GIVEN a plugin |
| * GIVEN that "Location/Plugin[name]" is the plugin absolute path |
| * THEN start should succeed |
| |
| - [ ] Scenario: Empty Location, plugin non existing path |
| * GIVEN config files with an empty location attribute |
| * GIVEN that "Location/Plugin[name]" is an non existing path (/libDoesNotExist.so?) |
| * THEN start should fail |
| |
| * * * |
| |
| ## Logging |
| |
| - info and warning |
| - Partly covered by the "set a logger" section |
| - [ ] Scenario: Plugin info and warning log |
| * GIVEN a plugin that log a unique string as info and warning |
| * GIVEN a logger that store logs |
| * WHEN the record logger is set |
| * THEN start should succeed |
| * AND_THEN the logger should have stored the plugin specific info and warning log |
| |
| * * * |
| |
| ## Basic Domains & Configurations management |
| |
| - [ ] Scenario: create and delete an empty domain |
| * GIVEN a started parameter framework without domains |
| * THEN creating a domain with an arbitrary name ("domain1"?) should succeed |
| * (AND_THEN the domain list should contain the domain name) |
| * THEN deleting the domain should succeed |
| |
| - [ ] Scenario: create and delete a configuration |
| * GIVEN a started parameter framework with a domain |
| * THEN creating a configuration in this domain with an arbitrary name ("config1") should succeed |
| * (AND_THEN the domain configuration list should contain the configuration name) |
| * THEN deleting the configuration should succeed |
| * (AND_THEN the domain configuration list should not contain the configuration name) |
| |
| - [ ] Scenario: delete a domain with configurations |
| * GIVEN a started parameter framework with a domain and several configuration (3?) |
| * THEN deleting the domain should succeed |
| * (AND_THEN the domain list should not contain the domain name) |
| |
| - [ ] Scenario: delete all domains |
| * GIVEN a started parameter framework with several domains |
| * THEN deleting all domains should succeed |
| * (AND_THEN the domain list should be empty) |
| |
| - [ ] Scenario: rename a domain |
| * GIVEN a started parameter framework with a domain and several configuration (3?) |
| * THEN renaming the domain to an arbitrary name should succeed |
| |
| |
| ### Error cases/special cases |
| |
| - [ ] Scenario: create a domain with an already-existing name |
| * GIVEN a started parameter framework with a domain (arbitrary name) |
| * THEN creating a second domain with the same name than the first should fail |
| |
| - [ ] Scenario: create a configuration with an already-existing name |
| * GIVEN a started parameter framework with a domain |
| containing one configuration (arbitrary name) |
| * THEN creating a second configuration in this domain |
| with the same name than the fist should fail |
| |
| - [ ] delete a non-existent domain |
| * GIVEN a started parameter framework without domain |
| * THEN deleting an arbitrary named domain should fail |
| |
| - [ ] delete a non-existent configuration |
| * GIVEN a started parameter framework one domain |
| * THEN deleting in this domain a arbitrary named configuration should fail |
| |
| - [ ] list configuration of a unknown domain |
| * GIVEN a started parameter framework without domain |
| * THEN listing configuration of an arbitrary named domain should fail |
| |
| - [ ] Scenario: rename a domain to an already used name |
| * GIVEN a started parameter framework with two domains |
| with arbitrary different names ("domain1" and "domain2") |
| * THEN renaming first domain ("domain1") to the name of the second one ("domain2") should fail |
| |
| - [ ] Scenario: rename a configuration to an already used name |
| * GIVEN a started parameter framework with one domain containing two configuration |
| with arbitrary different names ("config1" and "config2") |
| * THEN renaming first configuration ("config1") |
| to the name of the second one ("config2") should fail |
| |
| - [ ] Scenario: rename a domain to its own name |
| * GIVEN a started parameter framework one arbitrary named domain |
| * THEN renaming the domain to its own name should succeed |
| |
| - [ ] rename a configuration to its own name |
| * GIVEN a started parameter framework a domain and one configuration |
| * THEN renaming the configuration to its own name should succeed |
| |
| * * * |
| |
| ## Domain modification |
| |
| - set/get sequence awareness |
| - add/list/remove elements to a domain |
| - *(split domain ?)* |
| - *(listBelongingDomains ?)* |
| - *(listAssociatedDomains ?)* |
| - *(listAssociatedElements ?)* |
| - *(listConflictingElements ?)* |
| - *(listRogueElements ?)* |
| |
| ### Error cases/special cases |
| |
| - add (or remove) a non-existent element |
| - add (or remove) a parameter to (or from) a non-existent domain |
| |
| * * * |
| |
| ## Configurations modification |
| |
| - set a configuration rule; get it back |
| - trivial rules, complex rules (fuzzing ?) |
| - clear a rule |
| - save/restore a configuration |
| - set/get element sequence |
| - set/get a parameter for a given configuration |
| - set a parameter belonging to a domain and save a configuration |
| |
| ### Error cases |
| |
| - set a wrongly-formatted rule |
| - set a rule using an unknown criterion (or unknown criterion value) |
| - set/get a rule to a non-existent domain (or configuration) |
| |
| * * * |
| |
| ## Criteria |
| |
| - create a criterion |
| - exclusive |
| - inclusive |
| - *(list criteria)* |
| - set a criterion; get a criterion value |
| - exclusive criterion |
| - inclusive criterion |
| - special case: "none" value |
| |
| ### Error cases/special cases |
| |
| - duplicated criterion name (with the same type/with a different type) |
| - duplicated criterion value |
| - max supported number of criterion values |
| |
| * * * |
| |
| ## Settings Import/Export |
| |
| - import xml domains |
| - w/ or w/o settings |
| - from a file/from a string |
| - all/single |
| - export xml domains |
| - w/ or w/o settings |
| - to a file/from a string |
| - all/single |
| |
| ### Error case/special cases |
| |
| - invalid settings |
| - invalid xml (should we use a fuzzing framework ?) |
| - valid xml but not matching the XSD (ditto) |
| - valid but unusable data: |
| - unknown criterion/criterion values |
| - unknown parameters |
| - unreadable/unwritable file |
| - single domain import overwrite |
| - fail if overwrite not requested |
| - succeed if overwrite requested |
| |
| * * * |
| |
| ## Parameters |
| |
| - structures are correctly instantiated |
| - parameters in the "instance definition" |
| - component types instantiation |
| - parameter blocks |
| - bit blocks |
| - arrays |
| - check parameter mappings are correctly applied |
| - at all levels possible (subsystem, component type, component instance, |
| block, parameter...) |
| - context mappings, amends, instantiation mappings |
| - set/get a parameter |
| - parameter handles |
| - test on parameter values |
| - booleans |
| - bit blocks |
| - numeric types ranges |
| - fixed-point tests |
| - string length |
| |
| * * * |
| |
| ## Configuration application |
| |
| - apply configurations |
| - sequence aware domains |
| - forced sync |
| |
| * * * |
| |
| ## Misc/to be classified |
| |
| - tuning mode |
| - auto sync |
| - remote interface instantiation |
| - value space (raw/real)/format (dec/hex) |
| |
| ### Error cases |
| |
