Model-based testing (or MBT) has recently emerged as a popular approach to transforming and optimizing application testing. MBT offers significant benefits such as improvement of test coverage, reduction of testing effort and improvement of quality.
In my previous blog, we briefly discussed integration of MBT into the Continuous Integration (CI) process. Since then, some customers have asked me for more details on this subject. Specifically: how do we keep the model in sync with the requirements and code, and how we do incremental model development (like we do with code). In this blog, I will provide some of my thoughts and insights into how we make this happen.
I will use the capabilities supported by CA’s Agile Requirement Designer (ARD) as a reference for exemplification of concepts — similar capabilities exist in other MBT tools as well.
MBT in the CI Process
The following Figure (adopted from here) summarizes the basic process for CI.
The CI process is well understood, so we won’t elaborate on that here. We will discuss how MBT fits into the different stages of the CI process.
Backlog is typically captured in the form of user stories, feature descriptions or even defect reports and change requests. In MBT, the backlog items translate directly to corresponding models that express the flow (or the intended system behavior) associated with the backlog item. This is a key supporting principle for visual model-driven BDD (or Behavior Driven Development). For example, a user story for “login” functionality is described by a simple behavioral model for the login flow as shown in the following Figure:
The benefits of model-based requirements are described in my previous blog. The key issue for this discussion is how we maintain traceability between the backlog items and the models. This can be done in one of several ways depending on the tools being used for backlog management (such as Jira or CA Agile Central). Examples include:
- Embedding a link to the model from the backlog item – either in a source code management system (SCM) or in CA-ARD Server
- Embedding a link to the backlog from CA-ARD
When the backlog item changes (and is versioned), the model can be changed and correspondingly versioned as well – either in CA-ARD Server or in the SCM system. In addition, automated change impact analysis (due to change in model) can be performed that greatly improves productivity of both developers and testers.
Coding or Development
As described in my prior blog, model-driven BDD can be used to generate BDD and ATDD test cases that developers and SDETs use for unit and acceptance testing. The model can also be used to support generation of synthetic test data and virtual services to support the developer tests. The key issue here is how do we maintain traceability between versions of the model, code (and branches if appropriate) and other artifacts as shown in Figure below:
The following Figure captures the recommended flow for managing the traceability between the model and code.
The key idea is to perform incremental modeling (tied to each incremental user story for example), and then merge the model into the “mainline” model in sync with the code merge into the mainline branch. Just as SCM systems allow us to do diffs between versions of code to support the merge effort, we can similarly perform diffs between different versions of models in CA-ARD. In addition, we check-in all the artifacts generated from the model (such as tests, test data etc.) into the SCM system in sync with the code. This ensures that whenever code updates are pulled for builds by the CI system, the corresponding model artifacts are available for testing the build.
During integration, the CI engine aggregates commits from multiple developers to do an integration build, trigger appropriate integration tests, and if applicable, trigger packaging of the build into an artifact repository (or container).
The key issue here is enabling the CI engine to pick up all the tests assets (from models) that are applicable to the integration build. As described in the flow above, all changes made to code are tied to backlog items (in AMT tools such as Jira or CA Agile Central) which are tied to the corresponding models and tests assets. The envelope of test assets (called test manifest) tied to each change is already checked into the SCM system. The CI engine can pick up these test assets from the SCM system by correlating the test manifest data from the corresponding backlog items.
Summary and Looking Forward
Hopefully, this article has provided readers with some insight into how MBT can be integrated into Continuous Testing tied to the CI process. The end-goal of this approach is to enable what is called “In-sprint Testing”. We will discuss how MBT enables effective in-sprint testing in the next blog.