Let’s get real about test automation | Forte Group

QA automation rant: let's get real about test automation

I’ve got a litany of test automation pet peeves that I need to get off my chest. From the absurd expectations of those wanting to implement automation to the so-called test automation expert who thinks it will solve all their problems if someone hands them the proper framework to copy. Test automation is becoming an IT cesspool. Big, important, life-impacting problems are being solved every day, yet test automation seems to bring out the worst in everyone!

If this article saves you explaining to an IT manager why they can’t automate 100% of their SAP regression testing in three weeks (or ever) or the time to read one CV with the same bullet points copied across ten pages, it will be worth it. Hopefully, somewhere between the lines of the forthcoming acidic rant, there will be some educational tidbits.

About the author 

Lee Barnes has more than 25 years of experience working as a software quality and testing professional. So far, he has delivered test automation and performance testing ideas to over 500 companies. Lee sees his mission in closing the loop between development processes and testing and providing software testing services at a much larger scale. 

1. Managing Expectations

In his article entitled “Test Automation Snake Oil”, James Bach calls out some common but very reckless assumptions about test automation. I’ve encountered one more of these in almost every project I’ve been a part of: 

  • Testing is a “sequence of actions.”
  • Testing means repeating the same actions over and over.
  • We can automate testing actions.
  • An automated test is faster because it needs no human intervention.
  • Automation reduces human error.
  • We can quantify the costs and benefits of manual vs. automated testing
  • Automation will lead to “significant labor cost savings.”
  • Automation will not harm the test project.

You may think that the article was written long ago and wonder if any of this is still relevant? Yes, I know it was written in 1999 but it could have been written yesterday. The article is about principles, not practices and the principles are the same today as 20+ years ago. If this all sounds foreign to you, don’t worry. It’s understandable given all the marketing nonsense we’ve been subjected to over the years. However, if you’re fundamentally against what James is saying, you probably should not ask me to help you with test automation. 

Now, back to the rant. One of my biggest pet peeves is unrealistic expectations that flow from the reckless assumptions pointed out by James in his article. I’ve lost count of the organizations that think they have implemented test automation just because they’ve installed a tool. Was your system magically completed when you implemented Azure DevOps? Of course not, and it’s just as ridiculous to believe that this is the case for test automation.

While we’re on the subject of unrealistic expectations, let’s talk about the people that organizations ask to implement successful test automation. Just because you have the most skilled testers, business analysts, etc., does not mean you have a group of budding automation engineers. A vendor might insist their tool is “codeless” and can be used effectively by anyone who can operate a DVR but the reality is that the skills required to be successful with automation are pretty specific. 

Similarly, having the appropriate foundational skills does not guarantee success. I’ve seen many messes created by developers who have gotten their hands on a test automation tool. Being able to sling code and address technical challenges like dealing with custom UI objects is essential, but so is an understanding of software testing and what should (and should not) be automated. 

Should you automate testing?

Your QA automation readiness checklist, composed by QA experts.

2. Consultants and the “Perfect Framework”

I have limited patience for people who claim to possess a certain level of expertise and then wear out Google trying to find the magical one-size-fits-all automation framework they can drop in and take credit for. Testing forums are filled with posts like the one below:

Hi, I was wondering if anyone can point me to a test automation framework for Java/Selenium to work with a React application. If so, could you please share it?

Oh, a Java/Selenium framework for a React application? I have the perfect framework for you… and a nice bridge for sale in Brooklyn. 

I understand that these posts are coming from individuals with limited experience. We all need to learn – I’ve been involved in test automation for over 25 years and learn something new every day. But how about asking questions from a learning perspective? Asking for someone to give you an answer to a ridiculously broad and ambiguous question will not result in learning. 

Before you worry about a framework, you need to be thinking about why you’re automating in the first place. Are you trying to automate for the sake of automation, or are there business and development metrics you’re trying to improve? Do you understand the organizational, process, environmental, and technical factors you’re dealing with? 

If you know the answer to those questions, why not include them in your post and start a discussion around the appropriate approach to automation? This won’t result in a cut-and-dried solution, but it likely will result in a conversation that will get you thinking in the right direction and enable all parties to share ideas — and maybe learn something!

3. Percentage of Test Cases Automated

“What’s the industry standard for percentage of test cases automated?” is usually one of the first questions I hear when I’m meeting with new organizations. It’s tempting to respond with some exact number and then wait to see how long it takes them to realize I’m mocking them. However, I enjoy my job and would like to keep it, so I direct the discussion away from test cases, and towards testing activity in general. 

I’ve repeatedly found that automation’s low-hanging fruit exists in tedious testing activities like environment setup and test data creation and maintenance. Yet many organizations draw a line in the test case sand and blindly start marching towards it. 

Developing a test automation strategy is more important than achieving a specific number for test coverage. Take the time to understand the activities associated with your software development and testing efforts. Chances are there are opportunities to gain value from applying automation to very tedious and time-consuming activities that don’t include executing scripted test steps. 

The roadmap to implementing your test automation strategy

4. Automation ROI

Why do so many automation implementations have seemingly little value from a testing perspective? 

In my experience, there’s no overarching strategy driving the implementation. You wouldn’t build a business system by sitting a bunch of developers in a room and telling them to start coding. Yet so many teams take this approach with automation. If your test automation effort doesn’t further your testing objectives, it is worthless and will never deliver an ROI.

Test Automation ROI is always a major topic at test conferences and in white papers. Still, I rarely find anyone that can tell me IF they are getting a return on their investment, much less be able to quantify it. ROI is a calculation, and, by definition, calculations require inputs. If you’re not measuring your automation effort, you don’t know your ROI. Period. 

While we’re on the topic of ROI, let’s discuss the business cases that are put together to get approval to implement test automation solutions in the first place. I saw one earlier as part of a recovery project that compared the cost of the tool and training to the labor cost associated with manually executing a large regression test suite. Nothing else — no implementation costs, no maintenance costs — no costs to cover testing the stuff that wasn’t automated. And most importantly, no mention of the value provided by the manual testers. A first-year finance major would have doubts about this, and yet it was approved! I think we all know why this became a recovery project. 

We need to stop focusing only on the costs associated with testing when thinking about ROI. Let’s start thinking about the impact of freeing up your testers to focus on higher-value activities like exploratory testing. And most importantly, we should be measuring the impact of test automation on our ability to accelerate the delivery of high-quality software to our customers. 

Final word

Ironically, test automation is not only about testing. In fact, I think we would all be in a much better place if we did a better job of realizing that test automation is NOT software testing. Instead, it’s simply the use of tools, utilities, and technology, in general, to bring efficiencies to your software testing activities. The real ROI comes from the benefits derived from these efficiencies – faster feedback on new builds and, ultimately, faster delivery of value to your customers!

I’ll end with one parting thought: as we move forward as a community, let’s do a better job of thinking realistically about what test automation is (and isn’t). For now, if you’re thinking more critically about test automation, I’ve accomplished my objective. 

Test automation, scaled and customized to your business needs

Lee Barnes

by Lee Barnes

For the past 25 years, Lee has worked with organizations to implement improved software quality and testing strategies. He’s implemented test automation and performance testing solutions in hundreds of environments across a wide array of industries. Right now, he is a Chief Quality Officer at Forte Group.

Start a Project

Start here to accelerate or advance your business

Plan a Project

Answer a few questions and find the right software delivery model for you