In Metrics We Trust: How We Measure the Performance of Agile Teams

Introduction

At Forte Group I lead our team of software engineers, QA engineers, product managers, DevOps engineers and engineering managers. Our mission is to help clients accelerate their businesses with outcome-driven software development. We collaborate across the globe, with teams in the U.S., Europe, South America and Latin America.

In this blog post I'll detail how we measure the quality and performance of our Agile development teams. I'll take you inside debates we have about tracking story points versus issue count and explain why measuring both is the correct answer.

My goal with this blog post is to share what we've learned in the hope that it gives you a roadmap on how to measure and optimize the performance of your Agile teams.

Joint Performance Metrics for the Sprint Demo

While some client engagements involve a single Agile team from Forte Group, more complex engagements involve multiple teams working simultaneously. When we provide 2+ teams, we call such an arrangement a "scaled” Agile environment.

In scaled Agile environments we visualize the collective performance of all teams during the Sprint Demo session. This chart uses fictionalized data, but is the real-world framework we use to measure teams:

Group 1 (1)

You see three rows in the image above. We're tracking three teams here, so each row represents a team. The blue bars represent the team's commitments, while the green bars represent what was delivered. We track the ratio of committed to delivered. If a team committed to 50 story points but delivered 40, that's an 80% delivery ratio. We measure this ratio for both story points and issue count.

Story Points vs. Issue Count? Measure Both

In an Agile environment, story points and issue count are commonly used metrics, each serving a different purpose. You can have religious debates on which metric is more meaningful. In my opinion, it's not a matter of either-or. The answer is "both."

By measuring both story points and issue count, we can achieve a more comprehensive view of team’s work. Story points reflect only estimated effort, while issue count gives a rough sense of the total volume of work. Measuring both results in better understanding, tracking, and adapting to changes in the project. Additionally, teams can use historical data from both metrics to improve their commitment accuracy and can see where the trend is going to make more informed decisions in future sprints.

We find story points effective for:

Effort estimation
Capacity planning
Velocity tracking

We find issue count useful for:

Quantitative measure of total volume of "work" done in Sprint (all issue types - bugs, user stories, etc.)
Identifying items not being estimated for whatever reason
Identifying items carried over to the next iteration

How Many Sprints Are Needed Before Conclusions Are Drawn?

You can't draw meaningful conclusions after the team's first sprint.

Agile methodologies thrive on iterative progress and continuous improvement. It's important to view the development process as a series of interconnected sprints rather than isolated iteration. A single sprint may have outliers or unexpected challenges that can skew the metrics.

It's only over the course of multiple sprints that trends, patterns, and consistent performance metrics emerge. Waiting until several sprints are completed allows teams to navigate around early challenges and adapt to the iterative nature of Agile. This results in a more holistic understanding of the team's dynamics, capabilities, and areas for improvement, leading to more informed decision-making and sustainable success.

At Forte Group, our experience shows that six is the magic number.

In other words, the team needs to complete a minimum of six sprints before we draw conclusions. Even as initial conclusions are made, ongoing monitoring and adaptation are important, and the team should regularly reflect on their processes to make adjustments for improvement.

Should We Compare Story Points Across Agile Teams?

No, no and no. Absolutely not.

Story points are a subjective measure of the complexity and effort required to complete a task. Different teams interpret and assign story points differently based on their experience, skills, and perspectives. Comparing story points across multiple teams assumes a uniform understanding of the measure, which is often not the case.

Teams have different capacities, skills, and working styles, leading to variations of a story point “weight”. Directly comparing story points does not account for these differences.

A higher number of story points in one team does not necessarily indicate better performance if the project context is different. Comparisons should be contextualized within the specific project goals and requirements.

Instead of comparing story points across teams, consider this:

Challenge teams to deliver according to their commitment.
Watch the trend and drill down into details when you notice spikes or significant fluctuations of team’s commitment vs. deliverables.
Use story points as a relative measure within each team. Establish a consistent and shared understanding of effort within the team.
Teams should use their own historical velocity as a guide for future planning and commitments.
Instead of comparing story points, encourage teams to share their experiences, challenges, and best practices

Comparing story points directly across different teams can lead to misleading conclusions and is not recommended. I hope I made myself clear 😎

Conclusion

To measure the effectiveness of your agile development teams, having a data-driven approach is just the start. Once you're collecting data, the key is to determine the best way to interpret that data and make decisions from it.

Success in Agile is not just about speed but a predictable delivery with quality, according to the commitment. It's an ongoing journey rather than a destination, reflecting the core principles of the Agile philosophy itself. A project constantly evolves and changes, so we need to take a proactive approach, where teams embrace constant feedback loops, refine their metrics, and optimize processes to ensure continued success.

If you'd like to learn more about how we can deliver Agile teams to your organization, fill out the form on our Contact page and we'll be in touch shortly. Thanks for reading.