Test Parameterisation

Watch a video on test parameterisation here.

Overview

Test Parameterisation is a powerful practice to enhance your testing scenarios. The concept of parameters in testing is simply that your Test cases may define and use some parameters. During the execution of a test case the parameters are filled in with their actual value, so a parameterised variation of the original test case is produced.

So the main benefits of parameterisation are:

• By externalizing changing parts of tests as parameters it is easier to manage complex test cases
• You can automatically run multiple variations of each tests. If a parameter has multiple values provided that will cause several runs of each tests for each variations of the parameters.
• Parameters allows sharing information between multiple test cases. Sharing is easily achieved by defining parameter values in the test sets, which effectively and automatically shares these values in all test cases participate in the test set.

Using Parameters in Test Cases

Every test case can be parameterised by simply using parameters when editing the test case's definition. A parameter is defined by using the ${parametername} notation in the wiki text of the Test Cases. For example the "speed" parameter is used in the name field of this test:

That is it, use any name for your parameters, the parameter will be automatically recognised by CodeBeamer, and the "${parametername}" placeholder will be automatically filled in with the actual value during the run-time of the test.

Parameters can be used in the following locations:

• Name of the test-case
• Description of the test-case
• Pre- and post- actions of the test-case
• in "Action" and "Expected result" fields of test steps

Default values for Parameters

When placing parameters to Test Case definitions you can also provide some default value to the parameter. That is done using the "${parameterName|defaultValue}" format, so the "|" (pipe) character is used to separate the parameter and its default value.

For example the ${car|bmw} text defines a parameter named "car" and the default value of this is "bmw".

As you would expect the default value is only used if the parameter did not get any value otherwise. But if the value is provided in the container TestSet or in the TestCase or one of its parent TestCases then the default-value is ignored.

Empty string values in parameters

If a parameter has empty string value like " " (just one space) that is handled as if the parameter would be missing. This case the default-value of the parameter is used - if that is provided

Providing Parameter values in Test Cases or Test Sets

Assuming that you have used some parameters in your Test Cases definition(s) those parameters still missing some value.

The values of the parameters can be provided on each test case when editing it: in the "Test Parameters" section as screenshot shows below:

Values can also be assigned to a test set: the similar "Test Parameters" section also appear when editing a test set. These values provided on a test set will be shared and available in all test cases which are part of that set. (The exact parameter resolution algorithm is a bit more complicated than this, will be explained later...)

There is currently 3 ways for providing parameter values:

• As name and value pairs
• As a wiki table
• In an Excel sheet stored in Document Management

Providing Parameter values as name-value pairs

The simplest form of providing parameter values is entering it as name=value pairs in the "Parameters in Wiki" wiki editor. The following screenshot shows an example (e.g. speed parameter gets value "100"):

The rule is simple: each line in the wiki text is read as one parameter=value assignment if possible. Also visible on screenshot you can optionally put the values to a preformatted block (curly brackets {{{..}}}), but if missing those will be automatically added on save.

You can only define one dimensional parameter sets this way however. For multi-row parameters use the Wiki tables explained next.

Providing Parameter values as a Wiki table

Parameters can get values from Wiki tables. Just add a Wiki formatted table to "Parameters in Wiki" wiki editor, and the contents of that table will be used as parameter values. An example:

The values are taken from this Wiki table as:

• If there is a header row in the Table then the text in each header is used as the parameter names. If no header then the first row is used as parameter names
• The further rows below the table-header used as values for each parameter in each column.

Practically by defining a table you can define a multi-row parameter sets. This means that when running a test case it will be run multiple times: once for each value row in this table.

(A bit more complicated though, absolutely correct only if the test is using all parameters from the table, will be explained later...)

Providing Parameter values in Excel sheet in Document Management

The next way for providing parameter values is by selecting an Excel sheet from Document Management. An special ".codeBeamer/Test Parameters" directory is automatically created in the current projects' Document Management, where the Excel files can be uploaded to.

These files are then can be used as parameter data sources by picking them in the "Excel in Document Management" tab. An example selection:

The rules here are:

• Only the 1st sheet is used from the selected Excel
• First row of the Excel document is expected to contain the parameter names
• Further rows are used as parameter values

The benefit of using an Excel sheet and Document Management is that you can reuse and share this sheet between different test cases and test sets. Additionally the Document Management controls the access to the shared file, and manages versioning/history of that, so you have greater control over the changes.

Providing Parameter values dynamically from CBql or any other sources

See: Tip: using dynamic data as Test Parameters like cbQL results or any plugins' result

Sharing parameter values between Test Cases and Test Sets: Parameter inheritance

As mentioned earlier parameter values can be assigned to both Test Cases and Test Sets. As known the Test Cases and Test Sets can be organized to tree-hierarchies (i.e. any Test Case/Sets can have a parent and many children). These parent-child hierarchies of the Test Cases and Test Sets are used when looking up parameter values, as the "child" elements inherit the parameter values defined on higher levels.

Practically it is desirable that the functionally similar Test Cases are grouped below a parent/container Test Case, and/or the functionally similar Test Cases are grouped to one Test Set or hierarchy of Sets. If this is happened so then it becomes easy to share parameter values between them by defining these values on the parent elements.

How inheritance works: the search for parameter values

When a -parameterised- test runs the Test Runner will search for the missing parameters in these hierarchies bottom-up as:

• First checks if the current Test Case contains values
• Then scans all parent Test Cases of the current Test Case
• Then checks the current Test Set which is being run
• Then scans all parent Test Sets

In the following example the search for the parameter values is performed in this order: "A" TestCase, "B" TestCase, "Y" TestSet and "X" TestSet:

X (TestSet)
└── Y (TestSet)
    └── B (TestCase)
        └── A (TestCase)

This search stops the earliest when "some" value found for all parameters. (For example if all parameters got value by scanning the current Test case then no parents will be scanned at all.).

In this search it may happen that multiple parameter sets are defined at different levels of the scanned hierarchy. These parameter sets are kept and merged if necessary to produce a complete parameter-set. The details of that is explained in Advanced: Parameter resolution inheritance and merging

Reviewing and managing Parameter configurations

Once you start using parameters and provided few values the question starts to arise like:

• What are my parameters used this test case or test set?
• What are the actual values I've provided on this test case or test set?

The "Test Parameter" section shown on the detail page shows this information. The tab is available for both test test cases and test sets. An example screenshot:

This part contains:

• The number in the tab's header next to "Test Parameters" shows the number of unique parameters used in the current test case or test set.
• The "Used Parameters" section shows the:
  • unique parameter names (ordered by name)
  • Next to each parameter it shows if that parameter has some value already, the parameters with no value yet are highlighted in red . There the "Own" is displayed if the value is coming from the current test case/set. The "Inherited" text is displayed for inherited values which are defined in one of the parents'.
  • For test sets this list shows all parameters in all test cases in this test set. Plus for each parameter it lists those tests where the parameter is used. So that makes easy to find where is a parameter defined.
• The "Own parameter values" and "Inherited parameter values" sections show the actual values found either on the current test case/set or inherited from the parents.

Parameterised Test Running

When running parameterised tests the Test Runner will automatically resolve the parameter's values and will replace the parameter placeholders with the actual values. This means that the parameters are not any more visible in the text of the test, however the actually used parameters are displayed for the user. Following screenshot illustrates:

When the test case contains some parameters the "Using parameters..." side-bar will appear in the Test Runner. This displays:

• The currently resolved parameters with their actual values
• If some parameters' values are missing this will display a warning message with the list of missing parameters
• When multiple rows of parameters available then this will display how many executions of this same test will run to complete all parameter-sets

If a parameter's value is missing then the original placeholder is left in the text. For example if the ${speed} parameter has no value then the text "${speed}" will be kept in the test's text.

Providing missing parameter values during the run of a Test

Starting with codeBeamer 8.1.0 if a parameter's value is missing while running a Test then the Test Runner will ask for values for the missing parameters.

The following dialog appears, where the Tester has the option to fill the missing values:

The rules are:

• The user can enter value(s) for all missing parameters, or can also leave them empty.
• If a value is entered then the Test will be reloaded and the placeholder for the provided parameter(s) (like ${parameter}) is replaced with the new value.
• If the value is left empty then the original ${parameter} placeholder is shown
• The newly provided parameter value will be saved to the current Test Case Run, and will appear in the parameter list and also in the reports as the "ordinary" parameters
• If a missing parameter is filled that:
  • Will only be used for that single Test Case Run, and discarded after.
  • If the next Test Case Run misses the same parameter that will have to be entered again
  • The parameter value is not written back to the Test Case, but is used for this single Run. The reason is simple: that parameter may have been left empty deliberately because it can only be provided during run-time, and such run-time must not affect the Test Case definition which might be used several places elsewhere.

Running a test several times driven by parameter-values

The Test Runner will automatically choose and offer the next test-parameters by picking the first parameter-set which has not been run yet.

If the Tester wants to choose a different parameter-set manually then can select it by clicking on the "Select..." button in the runner.

Then an overlay dialog appears showing the Parameter-sets which has not been run yet. This is not possible to select already completed Parameters here.

This example shows 3 parameter sets available. The current Parameter-set is selected, but by clicking on some other and choosing the "Select" button the user can select a different one.

If you select a new Parameter here then the current Test results will be lost, because the test-runner will be reloading with the new Parameters inside: a warning dialog will be shown to pervent this.

Also the Tester can skip the remaining parameters completely. So the remaining parameterised runs of the current test case can be skipped by clicking on the "Skip All" button in the parameters' side-panel. This might be useful if there are simply too many parameters found or generated, like in case of 100s or many more parameters...

Skipping some Test parameters and Partly Passed result

When running Tests with parameters the Tester may decide to skip some of the parameters. That can be done by clicking on "Skip" button next to the Parameters' part inside the Test Runner.

When Parameters are skipped however that is indicated by a special result Partly Passed for the Test Runs. This means that the:

• Test was Passed: successful
• However the Test is parameterised and some of the parameters were not run by the Tester (those are skipped).

The Partly Passed result is handled similarly to the Passed result, however this is indicated on the TestRun page as follows:

The skipped parameters are remembered and a "Partial" badge indicates the partial results.

During the run you can still select and re-run previously skipped by parameter using the "Select..." button of the parameters:

Adding Partly Passed support to legacy Test Run trackers

• Go to Tracker customization page of the Test Run tracker. Find the "Result" field and click on "Options"
  
• Then add the "Partly Passed" option as here:
  

Reporting bugs of parameterised test case runs

When a test with parameters are run and a bug is being reported in the Test Runner the generated bug report will automatically contain the actual values of the parameters. This makes the bug reproducible.

A screenshot:

Reviewing Parameterised Tests' results

After a Test Run is complete the result of each parameterised tests can be viewed by looking at the Test Run's "Children" tab. This example shows when a test had 3 parameter sets and so 3 matching runs:

As shown on the example this "Run of Engine..." test had 3 parameterised runs. They appear below and as child of a container test-run which is automatically created when a single test case has several parameterised runs. This container test run issue summarizes the result of each parameterized runs, in our example it is "FAILED" because one of the parameterised runs are FAILED.

Additionally if you click on the result test-run of a parameterised tests you should see the actual parameters in the description of the test-run. This screenshot shows that:

Parameterised Tests and the Coverage Browser

As expected the Coverage Browser will show the result of parameterised tests too, however it will not show all parameterised test runs, but only the aggregated result of them. For example if the "Engine test" was run 3 times (because it had 3 parameter-rows), all was PASSED then the coverage will show one PASSED test. If however one of these 3 test was FAILED the coverage will show FAILED as well.

This example screenshot shows such aggregated result:

Advanced topics

Configuring existing Test Case and Test Set trackers for parameterisation

For using (=defining) parameters you don't need to do any configuration, by entering the ${parametername} placeholders to description or name fiels will work seamlessly.

For configuring parameter values you should add a new wiki field with the name "Test Parameters" to the Test Case or Test Set trackers configuration. This field stores the configured values (or the reference to the Excel sheet). Open the configuration gui, and add a new custom field like this:

The permissions of this field should be set that typically the "Test Engineer" and "Stakeholder" roles should have read-write permissions on this field, as they will configure the parameter values. The "Tester" roles should have at least read permission, otherwise the configured parameters won't be found during run of the test. Also if you want to use an Excel sheet in the Document Management then you should set the permissions of that Excel sheet so that all "Testers" should have read access on these parameter files.

The default permission configuration of the "Test Parameter" field looks like:

For Test Run trackers and Bug trackers there is no need for configuration. The parameters' values are stored in the "description" of these issues, so as normally these fields are writable by Testers that should work fine by default.

Parameter resolution: inheritance and merging

As described previously the parameter values are resolved by the Test Runner when running a test case. This is performed as:

• First the algorithm figures out which parameters are used in the current Test-case.
• Then the algorithm tries to find parameter values in this order:
  • Current test-case
  • Parent test-cases (bottom up)
  • Included TestSet: If the Test-case is inside of an included TestSet then scan the included TestSet
  • Current test-set
  • Parent test-sets (bottom up)
• The search for parameters is performed bottom-up (from the children towards the parents) continues until all parameters got some value, but stops immediately when the all parameters has are found.
• When during the search multiple but partially complete parameter-sets found then these are merged/joined together in order to build a complete parameter-set where all parameters are found.

The details of the resolution algorithm is:

Runner keeps only used and distinct parameter from the parameter-values

When this happens the parameter resolution algorithm will ignore and remove the irrelevant values, and only keep the unique/distinct values in order to avoid duplicated runs of the a test case with same parameters.

As an example imagine that the current test case is only using a parameter ${fruit}, and the parameter values found is this table:

Because the "city" and "price" parameters are unused in this test case and they won't appear in the test case anywhere they are irrelevant. So these values/parameters can be dropped from the parameter-set and so only 2 rows would remain both having the "fruit=banana" value. If we would run that it would cause that same test would run with same "banana" parameter twice, which would be quite annoying for Testers.

So to avoid this only the unique parameters-set combinations are kept. Essentially this works same way as SQL's SELECT DISTINCT statement.

Merging (joining) parameter values

When scanning parameter-values from the test-case and test-set hierarchy it may happen that partially complete parameter-sets are found on different levels. These parameter-sets are joined together to fill up the missing parameter-values required for running the test case.

Cross joining parameter values

Let's see an example:

• The current test case uses (so requires) 2 parameters: ${fruit} and ${city}
• This test case also defines these multi-row values:

• And the parent test-set defines these values:

During the scan for parameters the resolver find the 1st table, but it only contains values for ${fruit} parameter. These values are kept, the scan continues to the parent test-set, which contains the 2nd table, which contains several values for the ${city} parameter. Now the join algorithm does a CROSS JOIN to join these 2 tables in order to produce the possible parameter combinations. Effectively the test will run with 4 different parameter-rows produced by the cartesian product:

Natural (left outer) joining parameter values

The cross join performed is nice but may produce too many parameter-rows. However if the parameter-sets have few common columns then a NATURAL LEFT (OUTER) JOIN is performed similar way as SQL databases would do.

An example:

• The test-case requires 3 parameters: ${fruit}, ${city}, ${price}
• First parameter set found on test case:

• The second parameter set found on parent test set:

Then the resolution algorithm will do a join of the 2 parameter sets similar to NATURAL LEFT JOIN in SQL. This means that it will join the 2 tables using the common parameter (=column) names in the parameter sets, and for all rows in the 1st set it will join all matching rows in the 2nd. An "LEFT OUTER" join is performed, so all rows in the 1st set is always kept even when there is no matching row in the 2nd set.

The result in this case will be:

So the test case will be run 5 times for each row in this joined parameter set.

Limiting number of combinations produced by joins

As visible on the previous examples the JOINs may produce large number of combined rows. Therefore the resolution algorithm limits the number of rows joined to one "left" row to 10. So if one row in the "left" parameters has more than 10 matching joined row on the "right" (using natural or cross join) only the first 10 rows is joined and rest is ignored.

This helps to reduce the number of combinations.