Introduction
Quality assurance (QA) automation has quickly become a buzzword, and its use seems inevitable for enterprises of any size if you listen to industry leaders. They have a point – the 2020 GitLab Survey shows software testing remains the number one bottleneck, with 47 percent of companies identifying it as a reason for delays.
Meanwhile, test automation is on top of trends with the faith placed in it. According to The Evolution of Test Automation survey in 2018, organizations that automated more than half of their software testing efforts reported faster testing cycles (88 percent), and were able to catch bugs earlier (68 percent). So, does that mean you have to jump on the bandwagon and fill any gaps with automation?
There’s not a simple yes or no answer. While QA automation can help you release faster, it can also quickly burn through your budget and set you back. The key to creating a balanced test automation strategy is understanding the misconceptions that surround the topic.
Automation always meets deadlines
There may be a point in your testing journey when you realize that no matter how many experienced software testers you hire, your team is lagging behind and struggling to keep the same test coverage, and that releases become a race against time. The problem might be in accumulating regression debt: you can’t move forward without thoroughly retesting basic functionality, and old features tend to inflate your task backlog.
Your project duration may be seen as a litmus test to determine whether you need to consider QA automation. If your project is not a complex solution or you plan to complete it in less than 10 sprints, you can solve your issues by simply hiring more people instead of carving out some of your team’s precious time to automate tests.
Automation is expensive
It’s true: hiring quality assurance automation talent is expensive. Moreover, 58 percent of quality owners report that QA automation turned out to cost more than they initially expected. But what is also draining, both mentally and financially, is constantly managing your quality expectations despite having a skilled team and a reasonable testing scope.
There are three reasons why automation costs this much: the engineering resources, the test writing, and the environment maintenance. If your automated tests are not well written, or your developers don’t do a great job maintaining the testing automation suite, the automated tests will not provide you with great test coverage, and bug fixing costs will continue to go up. Keep in mind that the time spent on maintenance of the automation framework will overlap with the time you spend developing and testing new features. If you rely on your in-house resources only, you may experience severe planning issues later.
To sum up, sticking to manual testing is expensive all the way, while QA automation maintenance is cheaper from a long-term perspective. If your software testing goals are short-term, benefits like consistent release frequency or covered regression scope hardly apply.
How do you know test automation will pay off?
Test automation is not always worth it finance-wise, so it’s early to write manual testing off. There are cases one can’t test without human intelligence – for example, systems based on technologies like voice recognition.
Automated testing is all about reducing the time and concentration pressure that routine tasks put on testing specialists, and there are repetitive cases in every project. And don’t forget about the human factor. When you encounter massive data sets or APIs, sending thousands of manual requests and checking every response is near impossible.
— Dmytro Rainchuk, SDET (Software Development Engineer in Test) at Forte Group
Automation means predictable quality
Predictable quality shouldn’t be confused with a 100 percent pass rate. Predictability means that you know your next release won’t contain critical bugs which will disrupt the user experience or ruin business-critical workflows. Even though an automated test suite implies impressive scope, there’s no way hundreds or thousands of tests will pass every time. If it happens, this alone is a reason to be concerned and to double-check your automation suite quality.
On the other hand, a lack of any predictability results in the inability to release on demand and provide sign-off reports to the stakeholders before introducing new features. Without information circulating in a tight loop between software testing and development teams, your tests, automated or not, are a shot in the dark.
When you automate wisely, it allows you to cover the regression debt, handle complex cases manually, and provide feedback faster – all without seeking compromises.
How does QA automation make a difference?
From the software development perspective, I can say that there are basically no projects which don’t require some automation at some point. Quality is a priority nowadays, and automation makes quality more attainable than ever. Regardless of its costs, test automation provides a competitive edge since it accelerates product time to market. The one who delivers it first wins.
— Roman Kokitko, SDET at Forte Group
Automation has no limitations
QA automation doesn’t mean getting rid of manual tests, but rather using QA tools to redistribute the testing load. With enough effort and an unlimited budget, you can automate everything, but there are types of testing that are less effective when automated and cheaper to complete manually.
Remember that automated tests leave no room for imagination and exploratory testing; they only execute what you’ve put in them. If your product is centered around its look and feel, test automation is a poor choice. No script can evaluate visuals, usability, and UX the way human eyes do.
The biggest drawback of, and at the same time the biggest advantage of, manual testing is human involvement. Automated tests can’t predict the user’s reactions or follow complex and multi-step user scenarios. Instead, focus automated tests on possible load and performance issues – this is where the potential of automation is peaking.
When is test automation the only choice?
In general, automated testing is recommended if regressions are permanent and require the same scenarios to be tested over and over again, or there are a lot of end-to-end workflows. In such cases, multithreading can easily cut execution time even for very long test cases. And, if you perform cross-browser testing, automation is necessary.
Of course, manual testing will still be used in the future. A high percentage of manual testing is performed at least at the initial phase of most projects.
— Yuriy Zosin, SDET at Forte Group
Will manual testing stay or be replaced by test automation?
To summarize the answer to the QA questions testing folks strongly disagree on, we once again turned to the experts:
As always, the truth lies somewhere in the middle. The end of manual testing as we know it is nowhere to be seen. For now, companies are interested in hiring T-shaped specialists that have manual testing experience as well as coding and test automation skills. For anyone who wants to work as a QA engineer in the future, automation experience will be a benefit. At Forte Group, we make sure manual testing engineers learn the basic principles of building a framework and writing automated tests.
— Roman Kokitko, SDET at Forte Group
Key takeaways
- Automated tests are as reliable as the engineer who wrote them. After all, they are just more code to write and maintain.
- Using automation to ‘reduce’ manual testing is the wrong strategy to follow. Instead, think of test automation as a step towards improving long-term ROI.
- Use automation to redistribute your QA efforts in favor of testing new functionality without regression debt holding you back.
