Test environments can be a problem. They don’t have the simplicity of development environments, but they are expected to follow the same rules of agility. They get none of the respect and attention that production environments enjoy but are expected to mimic them.
According to The 2020 Continuous Testing Report put together by Sogeti and Broadcom, “36% of respondents spend more than half of their time building and managing test environments.” More than half! This means that for many teams, test environments are a major barrier to testing and to continuous testing.
If it takes you a week to set up an environment and a day to test, automating the test isn’t going to make you agile. It will only get you down to from nine to eight days of testing, at best. Below are some tips to eliminate the testing environment barriers to help you achieve continuous testing and speed time to market.
What are common Test Environment barriers?
Before we create a plan to ease test environment constraints, we should cover the common test environment barriers that impede innovation.
Lack of ownership
Not acknowledging the time that it takes to set up and handle test environments is one of the biggest problems for anyone that wishes to progress towards continuous testing.
Thinking that test environments are someone else’s problem—usually IT—is common. And sharing a static test environment with other teams, while producing meaningless test results—is a widespread strategy that leads to teams stepping on each other’s toes. Without taking ownership of test environments, there is no way to achieve continuous testing. Whoever automates the environments will take over quality—and it better be you.
Lack of expertise
Test environments are difficult. It’s easy to be frustrated when you need to deal with them. How can anyone automate all this on-premise equipment? Is everyone expecting you to be an automation or a cloud expert now?
Without a way to automate test environments, there is no way to introduce serious testing to the CI/CD pipeline, which means your DevOps could end up relying on what seems like automated testing, but is actually small bits of unit and functional tests that run on environments that don’t come close to representing production.
You know this cannot assure quality. But calm down. It’s time to come up with a serious plan.
6 Steps for Creating an Effective Plan for Automating Test Environments
Test environments are difficult, but there’s a good chance that your test environments can be better managed and automated—fully, or at least to a degree that would yield incredible results. It’s likely that the results of automating testing environments would be much more beneficial than trying to automate tests alone, while ignoring the fact that you can’t keep up with setting up environments for these tests.
Here are the 6 steps for automating testing environments.
1. Map the infrastructure
- What infrastructure do you use in your test environments? Who owns this infrastructure? If it’s not you—is it possible for you to get ownership or be more involved?
Is it possible to adopt an immutable infrastructure approach to your environments (i.e. dynamically set up all components of your test environments)?
- If it’s possible, then consider yourself lucky. This means it will be easier to automate and leverage public cloud resources if needed. It will also be straightforward to harness your automation efforts towards full blown CI/CD automation.
- If it’s not possible, do not despair. There is still a lot you can do for more efficient testing and continuous testing. If this is the case, you should map your environments’ bottlenecks including the resources that you will be forced to pool or share (for example, on-premise legacy systems, unique or expensive equipment or licenses, or heavy databases that cannot be automated). It is possible to dramatically increase the efficiency of such resources and automate many aspects of using them in test environments (for example, scheduling and managing conflicts, applying golden images and returning to a known state, and more).
2. Map the application and data requirements
- What applications do you need in your test environments (including your own and 3rd party components)? Is there a difference in configuration between environments? Can you come up with a finite set of configurations that all or most tests can use? Do you have access to the artifact repository?
3. Map the data requirements
- What are the data sets required for your test environments? Can you come up with a finite set of data sets that most or all tests can use?
- Do you need to use production data for your test environments? If so, does it require anonymization/masking or subsetting?
- Do you need large amounts of data in test environments? This is something that can hinder automation and require investment in specialized data solutions. If so, is this required in all your environments?
4. Map additional automation requirements
- What types of test environments do you need? Which ones would be the easiest to automate? Which ones do you use most often?
- Go over how test environments are provisioned today. Is there anything in this process that cannot be automated? Why?
- Do you have access and permissions to all the artifacts, images, packages, and systems required for setting test environments?
- What systems will you need to integrate to?
- Your CI/CD pipeline is one system that you are very likely to want to integrate with.
- Are there special security considerations in your test environments?
- Do environments need to be https secure?
- How do you handle credentials in test environments?
5. Building a Solution
- Who needs to use environments?
- Although you may aspire to fully automated DevOps, humans still need to use your environments. Whether it is for manual testing or for troubleshooting automated test results, make sure you understand who the end users of environments are and what would be the best way to service these end users.
- How are you going to control environment usage?
- Are there policies for how environments should be used? How would you limit environment usage? What policies can help avoid a bureaucratic “approval flow” for users to get access to environments?
- Whose buy-in do you need?
- Like other automation initiatives, handling test environments is highly rewarding, but requires some upfront investment. You will usually need executive or management buy-in, and it’s important to identify early in the process.
- What results would your organization and management want to see? You may have a strong Continuous Automation vision, but it’s vital to map the milestones that would get you there, and continuously prove value along the way. In order to do that, you need to understand the value that different people in your organization are interested in. For example, test managers may put emphasis on being able to easily get access to one-click test environments and fast reproduction of test environments, while DevOps management may be more interested in connecting test environments to the CI/CD pipeline; IT may be interested in proving test environment savings related to automation, and engineering management may care about speeding time to market with shorter test cycles. Environment automation has huge benefits—you just need to make sure you show the right benefits to the right people.
6. Choose your technology
- After gathering all the facts, you can make a technology choice. There are plenty of options, and the right choice for you depends on your mapped test environment requirements, as well as your skills, desired timeline, and budget. Don’t forget that automation and orchestration is only part of the test environment solution and would rarely be the deal breakers. Self-service and governance are equally critical for a scalable solution.
Hopefully these 6 tips will help you to make your test environments less of a problem, so that you can accelerate your company’s devops journey. Good luck!