Better Design of Test Packs = Better Test Coverage
Who doesn’t want to boost testing efficiency? But the more difficult question that follows is, “Where do I start and how do I get it done?”. One area of opportunity is what we call Test Case Optimization or the ability to achieve maximum test coverage with the minimum set of test cases. Yet when it comes to Test Case Optimization, it is very easy to claim the ability to generate “All Edges”, and declare victory that you have a high test coverage using this terminology when discussing with other stakeholders.
In reality, chances are, they won’t understand.
We need to design our test packs intelligently, so we can clearly articulate our confidence in our tests. So, on this blog post, I will describe techniques we can effectively use to design applicable test packs, yielding optimal test coverage.
Test Case Optimization – How it Relate to the SUT
There are various test case optimization techniques that testers can use today, aiming to generate the right level of test coverage. In my experience using CA Agile Requirements Designer, the test case optimization techniques that are most commonly used by organizations I had worked with are All Edges, All Pairs and All Possible.
With CA Agile Requirements Designer, you can also view the relevant test coverage metrics of each of these optimization techniques. These metrics are great especially if you understand what they represent. The test coverage metrics can be very abstract concepts and the true meaning will depend highly on what you have modeled and the level of abstraction.
Let’s walk through an example to explain this.
In this example, we have modeled the process of filling in some check out details. We have not included any negative test cases.
Using All Possible paths on this model will generate a test set detailing every possible (valid) way of filling in the checkout details.
On the other end of the scale, All Edges would produce an optimal set of test cases covering, at least once, each piece of data/functionality such as shipping method, payment method and card type.
All Pairs provides a half way house between the two. Here, we have covered every pair of data/functionality. One pair could be using next day delivery and ordering with a MasterCard, another could be entering different shipping details and paying with a purchase order.
Now that you understand the test coverage, you can select an appropriate test case optimization technique. For example, If you are creating a regression pack, then All pairs could be too costly but choosing All Edges would not be a bad idea as you know you’d be covering the functionality at least once with minimal test cases.
Filtering and Pinning Path Generation
Once you have mastered the test case optimization techniques, you can use pins or filters to focus your testing for better test coverage. Do you want to test only the negative test cases? Is there a high risk area of functionality? Use the filters and pins to select areas of interest and then use the optimizations to generate the test cases you want.
Pin blocks and/or edges in the diagram to highlight areas of interest. For instance, the Discover card edge is pinned because we want to generate a set of test cases to exhaustively test this process. To pin a block/edge simply right click and select Pin It and then the colour you want to pin it with.
To generate a set of exhaustive test cases we will want to generate All Possiblepaths but this time we’re going make sure it only includes the blocks we have pinned. To do this, we can go to path explorer and create a filter. The filter will generate paths which include red pins only. If necessary, we can add some Boolean logic to create more in-depth filters but for now this simple filter will do. More about filtering can be found here.
Before we generate the paths, we need to ensure the inclusive filtering option is selected; we only want paths which include any blocks/edges that satisfy the applied filter (Pinned Red). Now we are ready to generate some paths!
Using this method, we have generated an exhaustive set of test cases covering every possible way of ordering with a Discover card. Of course, we could do more with this filtering technique: we can generate sets of positive or negative test cases, we could highlight areas of high risk and generate only paths with critical steps or we could generate paths covering a specific set of requirements.
This is key to generating intelligent test cases to achieve maximum test coverage using CA Agile Requirements Designer.
Visualize Your Test Coverage
When you have generated any test set it is important to check the test coverage metrics say what you are testing and what you are not.
For any set of test cases, you can view test coverage metrics by selecting the coverage icon. In this screen we are given metrics relating to four of the test case optimization types.
For the set of paths generated above we can see that there are test coverage metrics less than 100%. This is because we generated a test set with filters included. To see what we are not testing click the more button. Here, we can see which edges (functionality) is not being covered by the test set and conclude if this is acceptable or not.
Manual Path Creation
Manual test creation will always be a critical tool in any testers arsenal: tests of a critical nature must be included into a test pack; simple smoke tests are required before more testing and test coverage can be boosted by adding test cases.
To add a manual path to your test set, click the Add manual path button and simply drag and drop the test steps (blocks) in the order desired to create your test case.
If there are sub flows or decision blocks in your path, CA Agile Requirements Designer will automatically ask what path/row you would like to take for your test.
For more information on what has been covered in this blog please refer here in the documentation.
CA offers comprehensive solutions that automates the most difficult testing activities – from requirements engineering through test design automation and optimization. Start your Free Trial of CA Agile Requirements Designer today and fast track your Testing at the speed of agile.
Naomi is a Product Marketing Manager at BlazeMeter, part of the Continuous Testing Business Unit.