CONTINUOUS TESTINGCommon Hurdles to Continuous Testing, and How Your Organization Can Overcome Them

What is Continuous Testing, and What challenges can you face when implementing it in your company?
Avatar Damen GillandAugust 27, 20198 min

What is Continuous Testing?

Continuous Testing is the process of integrating automated testing into all parts of your software delivery lifecycle. By building automated tests that can be run at every step, you empower the people building the code to find issues before they cause problems later. Continuous Testing is the next logical step once you are implementing Continuous Delivery. If code is continuously being pushed to production, that code must be tested to ensure quality (and sanity).

While most organizations are aware of the value, and strive for Continuous Testing as an extension of their agile methodologies, implementation isn’t always smooth sailing.

Read on for some of the common hurdles that companies face and find out how to help your team avoid pitfalls and build momentum.

Common Hurdles to Continuous Testing

1 – Inadequate Requirements

Success or failure is largely determined by planning. If you go to your requirements tracking tool and find a lot of tickets with just a placeholder title in your “in progress” column, you know you’ve got a problem. Unclear requirements cause everyone to spend time in meetings trying to figure out what they need to build, and often leave developers guessing at requirements from the upstream stakeholders who aren’t involved in the day-to-day delivery cycle. All of this leads to dramatic inefficiency and, predictably, a high rate of failure.

The good news is you are not alone. Most organizations have poorly documented requirements, and this is this is easily fixable with some key process changes.

Suggested Actions

  • Requirements don’t have to be ultra-complicated. A user story, a list of acceptance criteria, and a few instructions for how to test will do the trick. (Note, testing instructions are especially important if the functionality requires any new data or integrations with other systems.) Adopt requirements standards or templates that have been reviewed and approved by the entire team.
  • If a certain framework is required, accessibility standards have changed, or there’s a new legal review process, make sure these things are communicated up-front to the entire team.
  • Ask for feedback at each step of the process. Make sure the team doing the work has time to review and get responses on all their questions before assigning it to them to work on. This can be as simple as inviting a broader audience to your backlog sessions, or adding a column to your board to ensure that things have been through proper planning steps before they’re moved to “to-do” status.
  • Ensure the entire team has sufficient training on what the process of inception to launch looks like. Publish your “getting to done” workflow and communicate it. Everyone in the organization should know the steps it takes to get an idea shipped and supported.

2 – Siloed Communication and Tools

Silos doom modern projects. It’s imperative that you foster a collaborative atmosphere between your teams. Work to break down the barriers stopping your developers talking to salespeople, or your testers talking with design. Responsibility for quality lies on everyone, so work to eliminate any “one-way” information flows. It takes a village to build good software.

Suggested Actions

  • Allowing everyone access to the same tools is key. It’s asking for disaster if your team is still documenting requirements in emailed spreadsheets or PowerPoint decks. Use a robust requirements tracking tool with the ability for documented conversations to occur asynchronously — such as Jira, Rally, or GitHub.
  • Open your tools and offer training to your entire team rather than just developers. This will give your devs direct access to requirement creators and stakeholders, reducing the amount of time it takes them to understand the intended goals of the requirements. Your development team should also be able to test their own code. Leaving testing solely to QA will create bottlenecks and reduce productivity.
  • Have open standups and post meeting notes. If there are blockers, document what they are and who needs to fix them, and notify the larger team so everyone knows what’s going on and can help with a resolution.
  • It’s easy to complain, but remember that praising someone for doing a good job tends to ensure they’ll keep doing whatever it is you like. Praise can also be a great way to call out actions that need to be adopted as best practices. Never miss an opportunity to give praise – especially to people on other teams. We don’t just want to improve code, we want to stay on the lookout for ways to improve processes.
  • Work to improve your release note generation process. Reaching the point where you can easily generate a summary of what’s in the release notes is a good sign that things are moving in the right direction. Mozilla does a great job on the Firefox release notes — if you need inspiration, try breaking it down into New, Fixed, and Changed functionality lists.
  • Do regular cross-functional retrospectives and set aside time to fix issues that are uncovered — after all, if nobody has time to make changes, nothing will change. Valuable insight can come from anyone in your organization. If you’re finding it’s hard for everyone to make it to your retrospectives, try varying the times, or set up a “suggestion box” – you can use a Google Form and easily capture responses from team members in other time zones.

3 – Your Team Hasn’t Embraced Automation 

Manual testing is incredibly costly. Additionally, given that testing is often time-boxed, it’s not likely you’re manually testing everything every time you move code between environments or do a release. This is very risky as you are pushing code that isn’t fully tested to your customers.

By planning for time to build out automated tests, you’re saving your organization time and money, as well as reducing risk. By some estimates, fixing a production bug is 30x times more expensive than fixing a bug during development. Also keep in mind that customer trust in your brand is very hard to restore. In terms of ROI, that’s going to make testing one of your highest returns.

Suggested Actions

  • Work to figure out what you need to test, and what automation coverage you currently have in place. Then make sure those test scripts and tools are shared with everyone in the organization. It doesn’t make sense to limit performance testing to the operations group alone — everyone should have access to all tools. And remember, if you have to do something more than twice, the tools are getting so easy that it will probably have been faster to have spent the time automating it.
  • Make sure you’ve got realistic test data and aren’t just looking for positive results. Be wary of using “always true” stubs for generating mock API responses, and allocate time to build out negative tests and chaos testing, as these are key for spotting issues before they hit your customers.
  • If you don’t have an automated way of generating good test data (that is, either pulling production data and sanitizing it, or generating realistic fake data) make it a priority to build it. Without quality data, you can’t get accuracy from your tests.
  • Think about using tools like Lighthouse to get a jump start on best practices if your organization is struggling with requirements. While this is for web projects, there are similar automated scanning tools for a lot of popular frameworks and platforms.

4 – Systemic Technical Debt

There’s no such thing as “future-proof” and it’s likely that nothing on the planet would be built the exact same way today as we built it 20, 10 or even five years ago. This is the nature of technical debt and every organization struggles with it. Ultimately, we’ve always got to move fast inside the obstacle course that is legacy systems, third-party APIs, and antiquated code.

Suggested Actions

  • Audit deployment workflows, and ensure you know how everything across all your interconnected systems is pushed to production. Work to ensure the environment provisioning and build process is fully automated and set up in such a way that any developer can rebuild the system as needed. The modern approach to “Can you make a build in one step?” has to include environment provisioning and test data generation.
  • Always keep your software dependencies up-to-date. This will reduce the risk of security vulnerabilities, and generally lead to more performant code. Keeping up to date is almost always safer than running legacy code.
  • Ensure that you have published coding standards. Every time someone updates a legacy system they should be bringing the code up to your modern standards.
  • Build out sufficient automated tests. The more tests you have, the more confident you will be in making necessary changes to your code base.
  • If you have multiple squads, make sure they’re all doing code reviews with each other. Ideally, you can switch your staff members around to allow developers who work in operations to rotate into production and vice versa. This lets everyone see a more comprehensive view of the application and understand how any corner-cutting impacts others across the organization.
  • Consider setting aside sprints to pay down tech debt; limiting the work to optimization, refactoring, and improving automation. It may sound scary to sacrifice new features for one out of every three to five sprints, but remember the old saying, “Slow is smooth, and smooth is fast.” When you clear out the hurdles, you will move faster.


The best way to ensure that Continuous Testing is a success for your organization is to work on identifying and eliminating dysfunction. Be proactive with requirements; be consistent in your approach; work to share tools and knowledge across teams; use the right data for testing; give everyone the opportunity to get involved by giving feedback and praise; and finally, be kind to your future selves by budgeting time for improvement.

Building software is a complex process; it’s never perfect, and often we just have to do the best with what we’ve inherited. But there is a better way. It won’t happen overnight, but by taking the time to implement positive process changes, you will increase your momentum and chances of being successful. Be patient, and look forward to reaping the rewards.


Damen Gilland