INTELLIGENT AUTOMATIONTest Automation for Managers: Setting the Right Expectations and Leveraging the Outcomes

What are the requirements for successful test automation for managers? How can managers automate processes in a maintainable and scalable way?
Albert Gareev Albert GareevApril 18, 20196 min

Automation sounds a bit like auto-magic: getting rid of manual effort while getting things done. However, industrial automation, that revolutionized production and our society, is very far away from automation of intellectual work. Some formalized tasks – intellectually mechanical, one might say – such as computation or syntax checking, were successfully automated. But whenever human judgment, exploration, investigation, or trial-and-error is required mechanical copying of the process produces poor outcomes.

Automation isn’t just a nice thing to dabble in; it has become a differentiator for many companies. Technical stuff is supposed to learn scripting and tools. But how can managers successfully leverage automation without getting hands-on with it?

Let’s review an example of a business process involving upgrade of a multi-million dollar trading platform. Before a new version could be deployed into production, a thorough and intense assessment had to be conducted to ensure that thousands of business rules and conditions are handled exactly the same as before.

Here’s a high-level description of the business process testing strategy.

For Day One of the simulation we designated about 3,000 transactions and some 500 corrections. For Days Two and Three we intend to put 2,000 more transactions, some new, some corrections and reversals. On Days Four and Five we put about 500 transactions for each.

All transactions must be entered during the designated operational window. Then batch processing jobs will be run overnight.

Every day Business Analysts investigate the logs and behavior of the system, that is responding to the trades entered. Each Foreign Exchange trade affects market rates for the subsequent transactions.

The simulation runs on two environments in parallel, to compare the behavior of the new version with the one currently used in production.

The idea to employ automation was on the surface. Different vendors promised similar benefits: saving time, reducing manual effort and errors, and so on. Fast forward – one of the enterprise automation solutions was selected and a suite of automation scripts was created.

This is when the team hit the wall. This is when I came onboard as an automation consultant.

The situation looked the following.

Two suites of test automation scripts, one for the existing system, and one for the system to be deployed, were created and run according to the business simulation scenario.

But scripts had failures. Intermittent failures, so the same transaction could succeed in one environment and fail in another. When run individually, scripts did well. But executing thousands of data sets was always erroneous. Furthermore, business analysts found themselves spending more time investigating script failures rather then investigating behavior of the trading system.

The first set of hard requirements for automation emerged.

Test Automation must be reliable in execution.

In case of failures, detailed logs must be available in order to help investigation and to help solving the root cause. Reliability includes robustness – ability to withstand fluctuations of different environments.

Test Automation must be maintainable.

Maintenance involves debugging and testing. Then there’s the updating of input data and expected results, and keeping up-to-date interaction points with the application under test, be that API or UI. For example, in UI automation, updating descriptions of UI controls is a significant part of maintenance. All of that together often comes at time and effort costs comparable to or exceeding the initial costs of creating automation. But, unlike the creation phase, maintenance is not expected to be so costly, and there’s simply not enough time to debug the scripts when execution results are urgently needed.

Test Automation must be scalable.

Running just one script adds little value. Automation must have the ability to run in a stable way while processing large data sets, and must include hundreds of scripts, when needed.

A manager, intending to implement and use automation, takes a responsibility of wearing two hats. On one end, the manager is a client who has a problem to solve, and some goals to achieve with automation. If the goal or goals were not achieved, the automation is useless.

On the other hand, the manager directs creation of the automation by setting the right criteria: reliability, maintainability, scalability.

After the right automation is created, it becomes a powerful tool, which can be applied for a great benefit or misapplied to the disadvantage of the business. Let me illustrate a case of the latter.

One day, my automation consulting gig started with somewhat unusual precautions: two pieces of ID, and a signed legal statement acknowledging that I will never disclose my login credentials or store them in any written form. That may sound a bit strong, but it totally made sense. A large, international financial organization processing millions of investment accounts must take information security measures with all seriousness.

My first objective was to evaluate the effectiveness and efficiency of an automation suite used for posting test transactions. Considering literally thousands of varying business rules, such a verification was indeed a difficult task. What did I find? The automation suite was stored on the corporate network and used by developers and testers. To log in to the applications, the scripts required users’ credentials be saved in a plain text file. Some of the users had accounts with access to the preproduction and production environments. Luckily, no one had yet taken advantage of this security flaw.

I implemented certain enhancements, making sure that login credentials were used only during the runtime and never stored in any form.

From this example, a new criterion for automation becomes obvious: security. No enhancements, no benefits from automation would be acceptable if security is compromised. Security is especially crucial demand when dealing with personal information and financial data. Even a minor breach or one-time occurrence are highly detrimental to the confidence of the customers.

5 Steps for Successful Test Automation

Automation is a service, a tool that may prove to be useful or turn wasteful. Aim to improve your existing business processes by using automation without introducing unbearable costs or risks.

When aiming to replace or augment existing process with automation, consider the following crucial aspects.

  1. Know Your Process

Remember to learn about your existing processes before trying to mimic them with automation. A bad process typically becomes even worse with automation.

Also keep in mind that in a big picture many processes look generic and repetitive. It’s important to look inside and identify which tasks require cognition and investigation, analysis and judgment, – those are inherently human activities.

2. Test for Reliability

Automation also can and should be tested. Only stable execution and verifiable results may promise the benefits from the automation of the processes.

Make sure to include reliability as a requirement from the start. On a technical level, reliability is achieved by using the right automation framework in a right way. Reliability can hardly be “baked in” when development of automation scripts is done.

3. Mind the Maintainability

Due to the very nature of the process, your automation suite is destined to be ever changing. Ability to update scripts successfully and quickly is a differentiator between benefiting from automation or writing it off as a sunk cost.

Maintainability, too, must be included as a requirement from the start. Same as reliability, maintainability greatly depends on the qualities on the automation framework.

4. Expect Scalability

No matter how good it is, a single automation script delivers very little value. But multiplied hundreds or thousands of times, this value might become very considerable. What’s important to keep in mind is that all drawbacks are also tend to be multiplied – and may destroy the value.

In automation, scalability requirements should include costs and efforts for adding more test cases, as well as executing and maintaining large automation suites.

5. Protect your Security

Automation brings new ways of data storage and interaction. And data in the modern world is among the most valuable assets. Assess existing security risks to make sure that automation doesn’t amplify any of them. Also make sure that new risks were not introduced.

By following these 5 steps, managers can create scalable, maintainable and secure test automation

Albert Gareev

Albert Gareev

Albert Gareev is a software testing craftsman and practice lead. With over 20 years of diverse industry experience, Albert spearheaded testing for complex and challenging projects. Key expertise areas include Agile testing, Automation, and Accessibility.