How we can set ourselves up for implementing Continuous Performance Testing? In this article, following the first part on four biggest test automation misconceptions and how to avoid them, I’ve laid out a relatively short pipeline to help with the conversation about performance testing and DevOps.
About the author
Lee Barnes has over 25 years of experience in the Software Quality Assurance and Testing field. He has been involved in implementing test automation and performance testing solutions in hundreds of environments across a wide array of industries. Most recently, he’s brought quality practices (with a focus on automation and performance) to DevOps teams.
Lee regularly speaks at industry conferences, including DevOpsDays, Agile+DevOps, STPCon, and StarEAST. Now, he has joined Chief Quality Officer at Forte Group to continue cultivating smart approaches to quality assurance practices across organizations.
Table of Contents
- Performance at the Commit Stage
- Performance at the Integration Stage
- Testing monoliths vs. Microservices
- Performance at the Acceptance Stage
- Putting It All Together
- Implementation Approach
- Performance in Production
- Key takeaways
The old traditional way of performance testing much less frequently leaves many possible culprits that require investigation. This e-book will go through each development stage where you can implement performance tests for your software product:
- Performance / Load stage. That’s where we will likely execute something that looks more like a traditional, system-level performance test.
- Integration stage. There it focuses on individual components or smaller groups of components. Tests that can execute quickly and have few dependencies (or have their dependencies mocked).
- Production. And not just from traditional production monitoring but using other techniques as well.
- And finally, we might be able to learn something about performance when code is committed, depending on what we’re doing.
Performance at the Commit Stage
There are some tools and libraries purpose-built for unit performance testing. The table here is something I gleaned from a research paper out of Charles University in Prague that lists the most popular performance unit test frameworks. Popular is a relative term here — they reviewed almost 100,000 Java-based projects on GitHub and found that only about 3% had some form of performance unit testing. Of those projects, the vast majority (about 72%) used JMH, which is part of OpenJDK.
Not every method in every class warrants performance testing — however, there is a benefit to testing performance at the unit level where appropriate. These are typically called “microbenchmarks.” These should be true unit tests where you’ve mocked the dependencies — you’re just running them with multiple threads and/or multiple iterations. Running them as part of the build process can be problematic because you don’t want other things going on in the environment to impact the results. However, frameworks like JMH can help you package your tests and run them on an isolated machine.
There are some essential considerations for including performance unit testing in your pipeline. Probably two of the most significant are controlling the length of the tests and how results are presented.
First, tests would have to run too long to get enough data to give a high confidence level in small performance fluctuations observed. Therefore, we need to understand that the tests will detect only noticeable changes in performance, which should be fine for most applications.
Second, the absolute value of the results is less important than the trend from build to Here’s a short video on the topic:build. Reporting an individual set of results back to developers is much less impactful than showing them how their last commit impacted performance. While the unit performance testing frameworks have assertions, I wouldn’t have your performance unit tests break the build — at least not at first when you’re starting to implement this practice.
Performance at the Integration Stage
Moving down the pipeline, we can learn about performance at the integration stage — after we’ve checked in code, unit tests have passed, and now our CI server is deploying a new build into our integration environment. Let’s take a look at what these tests look like.
Integration stage tests focused on individual components, which should be your most critical components (especially if you’re just starting). You can always scale once you’ve gained experience and worked out the kinks.
We are still early in our pipeline and want fast feedback. These are tests that include individual calls executed by a small number of concurrent threads for a small number of iterations. We don’t need to execute a real-life load at this point.
Depending on the maturity of your pipelines — for example, how well you control environment configuration and data — you may want to limit the complexity of your test setup by mocking dependencies of the component. You should also consider measuring system metrics by extending application performance monitoring (APM) tooling into lower environments.
Similar to performance unit tests, you want to execute these tests on every build that gets deployed here.Also similar to unit performance testing, the results are not necessarily deterministic, and you probably shouldn’t have the tests fail the build. Finally, it’s essential to understand that we’re not looking for absolute results here. As we discussed, we’re not putting a real-life load profile on the system. And this environment likely isn’t anything like production, so what we want to do is look for trends across builds.
Remember, we’re not trying to answer questions about how the system will perform in a production at this stage — instead, we’re trying to get feedback about the performance of a component so we can identify and address any detected issues as soon as possible.