TEST DESIGNAre you using the right test data in your test designs?

How to use test data management tools with test design tools to streamline your continuous testing process
Ruth Kusterer Ruth KustererJune 25, 20198 min

Most software bugs are not caused by faulty logic, but in reality through interactions between seemingly independent input parameters. Unfortunately, these interactions are often overlooked because that combination of values did not surface in your test data

Think of a date selector component: Today, you test the days selector, and you test your months selector, your tests will naturally pass, and you sign off on it. But the next day, a user enters “February 31”, and the back-end validation complains. Seen separately, both “February” and “31” are valid choices. But it’s when combined that this pair of values needs to be handled differently. It’s a relatively simple example, it does however make you wonder how many such interactions your tests have missed.

Not too many, not too few … Just the right number

Your existing test workflows today may rely on manually gathered test data to cover the known cases you wish to test. Often, this test data can come in the form of a spreadsheet, with the necessary permutations made of the relevant value combinations. Not only is this manual updating prone to human error, it is also a herculean task just keeping the test data consistent and up-to-date.

Obviously, it would be redundant to multiply out all possible combinations of parameters in an attempt to catch those overlooked interactions. But what if you could ensure that you at least tested all possible combinations of any two parameters in your data models? 

Combinatory testing methods such as All-Pairs testing (also called Pairwise Testing) lets you bring down the number of test cases to a minimum. For example, if you have four options with three parameters each, an exhaustive test like {A|B|C} * {I|J|K} * {P|Q|R} * {X|Y|Z} would require a prohibitive 3^4=81 test cases. In contrast, All-Pairs methods systematically discard all test cases that contain pairs of values that are already covered by other test cases—which brings down the number to a handleable nine:

Test case:Option 1Option 2Option 3Option 4

This is where continuous testing tools such as Test Data Manager (TDM) and Agile Requirements Designer (ARD) come in. 

ARD is a requirements-based modelling tool for which pairwise testing is second nature. We start by making it easy for you to describe your data model as a flowchart within ARD, with all parameter combinations spelled out. Each generated path from start to end through the flowchart corresponds to one unique test case. This is quite handy when you wish to associate test data with a data model. Since each generated path represents a series of choices made in the data model, you can use ARD to export flowchart paths as unique rows of test data.

Are you using the right test data in your test designs?
Associate test data within your data model…


Follow these steps in ARD to try it for yourself:

  1. First, start by modeling your data as a series of decisions in ARD.
  2. Next, right-click decision blocks and go to the Test Data tab, where you add a new variable, or just insert an existing variable.
  3. To assign a value to the variable, choose one of the following options:
    • Click Edit Native Value to define static values.
    • Click Edit Resolvable value in Data Painter to define dynamic values.
      The variable definition dialog in ARD will prompt you to connect to the Test Data on Demand service (see Test Data on Demand for details). 

test data design

In ARD, after you have installed Test Data on Demand, there are four types of test data that you can use when assigning values in a model: 

  • Static values
    Values such as numbers, strings, booleans, are pretty straightforward. You use them for non-dynamic test data, for example, server names and ports.
  • Algorithmically generated values
    Generating values algorithmically gives you more options: In short, you can tell ARD “give me a random string out of these characters”, “Replace @my.com by @example.net” or “give me a random date within that range”, and so on. 
  • Seedlists
    Thirdly, ARD can take test data from provided seedlists. Seedlists are stock data of the type that testers commonly need, but that cannot be algorithmically generated: Think of “one hundred French male first names” or “ten cities in Japan”. This generic test data is stored in tables in a database that is also used by TDM. 
  • SQL queries
    Lastly, you can write SQL queries and have ARD dynamically load data from your own test databases where needed. 

Each decision block can have several custom pieces of data associated with it, depending on the decision chosen in this path. A variable assignment in one block can overwrite an earlier assignment to the same variable; only the last assigned value sticks for any given test data value.

… And generate all pairs of parameter combinations

After you have associated test data with the model, you want to export all pairs of combinations of input parameters:

 1. In ARD, open the Path Explorer. Generate paths using the All Pairs optimization. Remember to click Store all paths. Close the Path Explorer.

2. Navigate to Test Factory in the menu and click Test Data. The Test Data window gives an overview of all test data assigned within a stored path.

3. Click the Resolve button now to preview the values if your test data contains resolvable dynamic values.  

4. From here, you export your generated data as a spreadsheet in CSV format. Or, if you are using TDM, export it to your TDM repository.

test data

Congratulations! You’re ready to use your generated test data in your custom workflows as before. But now, you can be certain that all pairs of parameters are covered, and not more (overtesting), and not less (undertesting). 

Take the next steps – Make the most of test data

From now on, every time you make a change to the data model, simply have ARD re-generate your test assets with the required coverage—and you can be certain that the necessary test data combinations are generated fast, consistently, and error-free.

Learn more about using different types of test data and how to reduce costly manual efforts. Join ARD expert Alyson Henry in her webinar on using Test Data in Agile Requirements Designer, where she will show you: 

  • How to overlay data combinations, dependencies, and business rules to ARD models
  • How each test case reacts to rich data sets
  • How to model applications with different data needs
  • And more.

Sign up here to Save your seat.

Ruth Kusterer

Ruth Kusterer

Part of a development team based in Prague, my interests lie in creating development and testing tools that take error-prone repetitive tasks off our shoulders and let our brains focus on the creative parts.

Related Posts