We recently hosted a webcast titled “Cost Avoidance & Optimization through Effective Performance Testing Strategies.” We also hosted “Performance Tour Live Event 2023,” an in-person event in Chicago. At both events I presented material with my good friend Scott Moore, Host of The Performance Tour & The SMC Journal. In this blog post, I provide a summary of what Scott and I covered.
We discussed:
- The true value of performance testing
- Understanding cost implications
- Building an effective performance testing foundation
- Shifting performance left and moving towards continuous performance testing
- Real-world case studies that inspire action
The value of performance testing
I like to compare it to an insurance policy. You can release mission-critical systems without performance testing, but you just don’t know what might happen in production. You could experience system failures and poor performance, which can lead to dissatisfied customers and lost revenue.
- Avoid costly outages
- Optimize resource costs
- Preserve brand value
- Increase user retention
- Increase customer conversion
- Improve employee productivity
- Improve search engine ranking
Scott noted that performance testing case studies are using data and measurement to help quantify its benefits. For example, on e-commerce sites, every second of reduced processing time can result in 7-10% more conversions (i.e., online sales).
How value is derived from performance testing
Next, Scott and I discussed how value is derived from performance testing. Starting from the top of the pyramid (diagram below), we progress through these steps:
- Findings
- Analysis
- Results
- Planning, Development and Execution
For the “Results” stage, Scott recommends that we state the results in tangible terms, such as cost savings or revenue gained. When you do that, you gain the attention of the C-suite, says Scott. I totally agree. Sometimes we can think about results in terms that are too technical and not focus on business results. We should start with the latter first.
An effective testing foundation
We can ensure value with an effective testing foundation:
The steps in the foundation are:
- Plan, Develop & Execute
- Results
- Analyze
Scott says that the planning phase is the most important phase. He encourages us to narrow the scope in this phase and figure out your “what and why.” Ask a series of questions to distill down the list of requirements and objectives.
Among these three phrases, Scott’s advice to performance engineers is to focus on Analysis. How you communicate the analysis and tell the right story to stakeholders helps you become more valuable to your company. Scott recommends that managers get their teams trained on analysis, as that will create a more effective team.
Continuous performance testing
Next we discussed Continuous Performance Testing.
I think of Continuous Performance Testing as the evaluation of performance at each stage of the delivery pipeline. It’s frequent and visible performance feedback that allows us to trend performance data across builds.
Scott and I presented these steps:
- Commit (automated unit performance testing)
- Integration (automated component performance testing)
- User Acceptance / Performance Load (system-level performance testing)
- Production (automated performance testing)
Scope of performance engineering
Finally, Scott discussed the wide scope of performance engineering. He says that it’s so much more than code. Performance engineering applies throughout the entire software lifecycle and starts before a line of code is written. It ends not with testing or production, but with the business actually using the software and deriving value from it.
Performance engineering sits in the middle and interacts with:
- Business & Contracts
- Dev
- Quality Assurance (Testing)
- Platform Engineering
- Monitoring / Observability
- Capacity Planning
- Simulation
