I recently participated in a panel discussion at Automation Guild 2024 titled "Continuous Automated & Performance Testing In DevOps" with my fellow panelists:
- Joe Colantonio, Founder of Test Guild (Host)
- Scott Moore, Founder of Scott Moore Consulting
- Paul Grizzaffi, QE Solution Architect at Nerdery
We had an engaging conversation about the current state of continuous testing. In this blog post, I'll share my takeaways from the discussion, including how I'm thinking about continuous testing right now.
What's The State of Continuous Testing?
Joe asked each of us to weigh in on how we're doing as an industry with continuous testing. I see a lot of variance across the industry, with organizations in different stages of maturity. On the "mature" end of the spectrum, organizations are using functional testing as the foundation, then layering on initiatives around performance, security and accessibility testing.
On the "less mature" side are organizations that take an old-school approach to continuous testing. They're trying to stuff traditional, large-scale performance tests into a pipeline without thinking critically about the scope of testing. This results in longer feedback loops.
Scott notes that companies who are doing continuous testing well are getting amazing results from it. Unfortunately, those doing it well are only 5% of all companies, according to estimates from the continuous testing vendors that Scott meets with. This is a disappointing number, but it doesn't surprise me. It happens when companies focus on a technology or tool, rather than taking a step back and figuring out how to deliver value more quickly to customers.
When you think about technology first, you end up implementing tools and tests in your pipeline, without an understanding of the overall objective you're trying to achieve. You're not thinking in a continuous testing mindset of what needs to happen to make things successful.
Paul agrees, noting that DevOps is more about a culture rather than a set of tools. In order to get that culture to where you want it to be, you need tooling and automation; however, just because you have a pipeline that runs your unit tests, it doesn't mean you've done DevOps correctly. It's more of a collaborative effort to reduce the friction between Ops and Dev and ship more economically and effectively. A lot of organizations haven't figured this part out.
How to Improve Your Continuous Testing Maturity
As Scott noted, only 5% of companies are in a mature state with their continuous testing. That means 95% of the industry is in a less-than-mature state. To progress along the maturity curve, I think organizations need to understand that DevOps is more than continuous testing, technology and CI/CD pipelines, it's a culture.
From a testing perspective, take a step back and keep the test objectives and risk analysis in mind. Take a close, critical look at the types of tests you have, and when you're executing them. Do you really need all the test suites you have? How can you maximize the value from the minimum amount of testing you're doing?
A big part of the DevOps culture is continuous improvement. So fail fast. Try something and if it's not what you want, tweak it. Or if it's early in the process, throw it out and go a different direction. Don't be afraid to experiment.
As for measuring and monitoring progress, stay away from counting metrics, such as the number of test cases automated or the percent of test cases automated. Counting metrics tell you nothing about the value of what you're doing. You end up automating just for the sake of automating. If you're moving the needle and meeting business objectives, then you can be quite confident that you have the right test automation program in place.
Finally, ensure that you're actually practicing Agile development. One reason so few organizations reap the benefits of continuous testing is that they're not really doing Agile development. Instead, they're doing waterfall development two weeks at a time. However much code they can produce in two weeks is what they accomplish, and there's nothing continuous about that. So if you're really using a waterfall model that's masquerading as Agile, your continuous testing program falls flat.
What Tests Are Best Suited for Continuous Testing Pipelines?
An attendee asked us this question:
What kind of tests are best suited for CT pipelines? Do you employ different tests or number of tests for different environments?
Paul notes that you first need to decide how long you're willing to wait for results. The more you're willing to wait, the more things you can test. For example, he notes that 40 tests taking one minute each will take 40 minutes. That can be too long to wait, so you decide to parallelize it. Well, that's going to cost you money, since you need to pay a cloud provider or run your own grid. You'll want to run a cost-benefit analysis to determine the speed you want in results, with costs considerations factored in.
I agree with Paul – you're balancing the need for fast feedback with the gain in confidence that you can move the builds to the next stage. You need to balance that confidence against the risk and your confidence in one environment will be different based on the context (e.g., injecting a drug into a patient vs. testing an icon in a social media app).
Conclusion
Kudos to Joe Colantonio for creating Test Guild and organizing such amazing content.
Wrapping up the panel discussion, one thought kept surfacing in my mind. And that's to recognize that performance testing is not a one-time concern. It's an ongoing, integral aspect of software quality. By fostering a culture that integrates performance testing seamlessly into our development pipelines, we not only identify and mitigate potential issues as early as possible, but also ensure that our applications consistently meet the evolving demands of users.
In other words, we move our testing initiatives higher on the maturity curve and get closer to the 5% of companies doing continuous testing effectively.
Let's make performance testing an essential component of our development philosophy and culture, resulting in more scalable and high-performing software solutions. Continuous performance testing helps us create software that exceeds the expectations of our users and stakeholders.
