The Test Suite Canvas: a framework for effective DevOps testing

Testing in DevOps must provide fast, accurate and actionable feedback.  It’s essential that organizations be very intentional about the automated test suites they include in their pipeline.  The test suites, and the tests within them, must provide valuable information that helps the organization assess the quality of their builds.  Additionally, the test suites must be capable of executing reliably on-demand to prevent testing-related bottlenecks.

Unfortunately, many organizations fail to achieve the effective test automation required to support continuous integration and delivery.  There are several common root causes, but they can all be traced to a failure to be intentional about implementing the test suites – i.e. organizations implement various automated test suites, but don’t put in the critical thinking and effort required to build and sustain solutions that can be reliably executed on-demand.

Test Suite Canvas

The Test Suite Canvas helps organizations understand what test suites are necessary and promotes critical thinking around implementing and supporting effective automated testing. By discussing and evaluating the eight dimensions of the canvas, organizations will get a clear picture of which test suites are valuable, and what is required to successfully build and maintain each test suite.

This version of the canvas was presented by Ashley Hunsberger at SauceCon 2018 and was modified from the Test Automation Canvas introduced in a blog post by Katrina Clokie.

To be successful in a DevOps environment, the entire team (or representatives in large organizations) should be involved in planning for their automated test suites.  This results in a much more well-rounded assessment of the dimensions of the canvas – and subsequently, a much more effective test automation function.

Test Suite Dimensions

Purpose

The purpose defines the business risk that is mitigated by the test suite.  Test suites that don’t have a clear purpose are likely not providing the appropriate value to the organization.

Dependencies

For each test suite, there are items that must be in place to support its creation and execution.  Dependencies might include specific tooling to create the test suite or hardware/services to host execution.

Constraints

Constraints are things that prevent the implementation of a test suite (or at least implementing them in an ideal way).  Along with dependencies, constraints must be addressed before test suites are implemented.

Pipeline / Execution

Execution frequency and the event(s) that trigger execution must be defined.  This provides important information for integrating test suites into the deployment pipeline.

Environment / Data

Environment and data instability are major contributors to ineffective test automation.  It’s critically important for the team to understand where automated test suites will be executed and how data will be managed in each environment. 

Ownership and Response

In a DevOps environment, automated test suites are counted on to assess the risk of builds as they move through the pipeline.  Test failures must be addressed quickly to ensure the team remains confident in the test suite.  It’s critical to understand who is responsible for both ensuring the test suite is in a working state and responding to failures.

Maintainability

Maintainability is a key component of effective test automation.  The team must understand the nature of future application changes (as best they can) and use that information to construct the automated test suite in a maintainable way.  In addition, the team must determine what minimum documentation needs to exist to allow a smooth transfer of ownership, if necessary, in the future.

Effectiveness

Implementing and maintaining automated test suites, while absolutely required for successful DevOps, is not trivial.  Therefore, it is critical for the team to continually measure the effectiveness of the test suite.  At a high-level, the test suite must continue to serve its purpose – effectively mitigate business risk.

Final word

Once the canvas has been completed for all potential test suites, organizations are likely to have a sizable list of items that are prerequisites to building effective automated test suites.  It’s helpful to add these items to a technical backlog so they can be tracked and worked as part of the organization’s ongoing efforts.

Going forward, it’s important to realize that automated test suites are not static entities.  Application and environment changes will impact the dimensions of the test suites and require them to be maintained.  Like application development efforts, a culture of continuous improvement should exist for test automation.  Without it, the test suites will decay and lose their effectiveness over time.