Implementing Performance Testing in a DevOps Pipeline

Client:
A global manufacturing company

Industry:
Manufacturing

Technology and Platforms:
LoadRunner (component and system level testing), Java / JMH (unit level performance testing)

The Goal

Establish performance testing activities in an existing DevOps pipeline

A global manufacturing company had transitioned to DevOps, but was struggling to include performance testing practices in their pipeline. The IT organization had an established performance testing Center of Excellence (CoE) that thrived as part of the waterfall model. However, the legacy approach to performance testing, which demanded long lead times and was focused only on end of cycle execution, did not integrate well with their new agile processes.

Forte Group introduced in-sprint performance testing and analysis activities with a focus on making performance part of everyone’s responsibility. Specific changes included:

Creating performance-based requirements.
Unit and component level performance tests as part of continuous integration.
Tracking trends in performance-related system telemetry across builds.
Utilizing service virtualization to remove system dependencies in upstream environments.
Using performance engineers from the CoE to mentor the development teams on key performance related practices and tools.

Top Obstacles

Outdated Approach

The client’s performance testing CoE required long lead times and code freezes during test execution that were incompatible with their DevOps approach.

Isolated Knowledge

Knowledge of performance testing and analysis had been isolated in the CoE for years, leaving the development teams with inadequate performance testing and analysis expertise.

Outcome

Performance Issues Detected and Addressed Proactively

The client was able to track performance trends at the unit, component and system level from build to build. Any changes that negatively impacted performance were detected in the sprint they were introduced. The result was an 85% reduction in customer-reported performance issues over the first 12 months of implementing in-sprint performance testing and analysis.

The development teams learned performance-related skills and were able to achieve the goal of making performance part of everyone’s responsibility.

Key Benefits

Production Performance Issues Reduced by 85%

The team was able to proactively identify performance issues when they were introduced versus waiting until the end of a release cycle. Performance-related production issues were reduced by 85%.

Shared Knowledge

Performance-related knowledge was disseminated to the development teams enabling performance to become part of everyone’s responsibility.