What the automotive industry can teach us about software delivery & QA

Among software services and solutions providers, “industry-leading” speed-to-market is a commonly parroted value proposition. The underlying promise of speed is to offer business value, sooner. But as we’ve discussed in past articles on agile methodology and accelerated software delivery, if you’re not building something that produces value, how fast you’re building it is irrelevant.

To illustrate this point, we need to look no further than to an industry that knows a lot about speed: automobile manufacturing.

How Toyota redefined the approach to quality

By the late 1970s, the once-mighty U.S. auto industry was faced with a choice: evolve or die. The ideas of famed management consultant W. Edwards Deming had helped elevate a once-inferior Japanese auto industry — in particular, the Toyota Motor Corporation — into an industry disruptor.

Deming’s philosophy was simple: Continuous, incremental improvements result in a quality product. By incorporating quality control principles throughout the production process, companies could produce the highest quality at the lowest overall cost over time. At the beginning of the post-WWII packaged goods era where the focus shifted to quantity over quality, U.S. business leaders initially rejected this idea. Rigorous quality control was viewed as overkill that slowed production time.

Deming’s focus on quality offers many insights on how we can build better software products that provide value, faster. Revisiting these ideas from the software development point of view offers great insight into how companies can improve their software delivery strategy. Deming’s work helped pioneer lean manufacturing and transform the auto industry. The same principles guided the beginnings of agile software methodology and, specifically, quality control in the software delivery cycle.

Why software product quality matters

Before diving into Deming’s theories, it’s important to discuss why having a quality product is important in the first place. We live in a time of rising customer expectations and declining brand loyalty. If a product doesn’t meet a customer’s quality standards, people move on to one that does. Even if the customer sticks around, low-quality products are burdened with an increased cost to fix and maintain.

How do you put continuous improvement into practice?

High-quality products don’t just happen magically. They are the result of rigorous testing, feedback, and corresponding adjustments throughout the product life-cycle.

Quality comes not from inspection, but from the improvement of the production process.

Dr. William Edwards Deming, an American engineer and author

Agile, DevOps and value-driven development: what’s the difference?

Since we’re throwing around several buzz-wordy terms, let’s briefly define what each of them means and where each fits within the larger picture.

Value-driven development is the vision that should guide what teams build next. Agile methodology empowers software delivery teams to smoothly execute their day-to-day activities with a clear goal (measurable value) in sight. DevOps is the culture and organizational structure that creates the optimal team to deliver on that defined value.

With each of the above practices in place, the next goal should be the creation of a “continuous” pipeline:

Continuous integration (CI): Code is continuously written into a shared repository
Continuous delivery (CD): Design, code, test, and release software in the shortest possible cycles
Continuous testing (CT): Testing occurs at each stage in the process with no downtime. If the code fails, it goes back to development; if it passes, it goes straight to deployment

Take a deeper dive: Video: Continuous Performance Testing in DevOps

Deming, Datsuns, and DevOps

Building a quality product, no matter the industry, is the key to both long-term growth and increased productivity. What elevated Japanese auto manufacturers from being synonymous with poor-quality Datsuns (see the 25-horsepower Datsun 113 below) to gold-safety-standard Toyota Camrys was continuous improvement — small, incremental changes to design and process over time. AutomotiveIndustrySoftware_Graphic3

Here are a few steps to implementing continuous software quality assurance for better processes, explained:

1. Marrying testing and coding with agile

In software, quality control practices occur through quality assurance (QA) and testing. Until fairly recently, the dominant software development model was the waterfall workflow, wherein development follows a linear path: design, code, test. In the waterfall process, each phase in the path can take months, creating an enormous gap between building and testing.

The waterfall approach is similar to those adopted by major American automakers — the one that Deming criticized: testing occurs after the product is built, meaning that any bugs or incorrect assumptions aren’t discovered until the final stages of production. This makes problems more difficult to detect and fix and can severely slow (or even halt) production.

Agile methodologies tighten design-code-test development cycles, usually into two-week sprints. This narrows the testing gap, but a gap of two weeks between building and testing is still too long. To fully incorporate quality control into the development process, testing and QA must occur at each stage.

This is best achieved by breaking down walls between departments and creating an environment where testing and coding information is freely shared throughout the team. In software delivery, it’s a feedback loop referred to as DevOps, the fusing of development and operations.

2. Eliminating barriers between departments through DevOps adoption

DevOps and agile are not competing ideas, but rather complementary ones. The agile manifesto lays out practices that enable cross-functional teams to iteratively deliver working software to users. All assumptions are validated by user testing in an environment that embraces flexibility rather than a predetermined roadmap.

DevOps methodology takes this approach one step further, creating a team structure that eliminates barriers between departments. In DevOps organizations, teams operate as a cohesive unit rather than disparate silos. Developers, testers, administrators, and even users work as a unified team that shares responsibility for building, testing, and deploying products.

Namely, at Toyota, each member of the team was responsible for the quality and outcome of the final product. This created a sense of ownership and pride in the car that rolled off the assembly line. AutomotiveIndustrySoftware_Graphic1

3. Testing is a continuous process, not a destination

As we’ve established, when testing occurs at the final stage of a production cycle, testing becomes a bottleneck that results in lower quality and longer production times.

That’s where test automation and continuous testing comes in. Building an automated testing platform requires an up-front cost that many organizations balk at. However, consider a construction analogy: If the product you’re building is a skyscraper, an automated testing platform is the scaffolding that must be built before a single brick is laid.

With that “scaffolding” in place, organizations can deploy code every 10 minutes rather than every two hours or two months. This produces feedback and results faster, making it easier to identify mistakes. Since testing can occur in real time, it allows the DevOps team to focus on developing new features and lower the cost of testing in the long run. Over time, the process streamlines, resulting in continuous, incremental improvements that create quality and efficiency — ultimately, a better product.

But to view automated testing as merely an efficiency and speed play misses the point. When mundane tasks like testing checklists are automated, the team is freed up to focus on a higher level of quality in development and customized manual tests. The results of optimized, automated QA and testing influence how the development team writes new clean code that can easily fit into the testing structure.

I have a CI/CD pipeline. Why is the quality still lacking?

Many organizations have adopted DevOps and a CI/CD pipeline but have yet to realize any benefits of quality assurance or valuable outcomes. Consider these questions when evaluating why a CI/CD pipeline isn’t producing results:

What are you trying to accomplish with a CI/CD system?
If the goal is to simply move faster, you’re not following the principles that made Deming and Toyota successful. A CI/CD system designed as a speed-to-market play will only result in poor quality. Fix the machine that builds the product— by optimizing your CI/CD pipeline for quality—and the result will be a better product with less waste in the process.
How are you automating to ensure software quality?
The goal of a CI/CD pipeline should be to expose problems as early in the process as possible. When built correctly, a CI/CD system won’t even allow you to check in the bad code. When an error is detected, halt production, fix the problem at its root, then move forward. With such a practice in place, developers will become better at making sure whatever error occurred doesn’t happen next time. Master the process first, then automate it.
Does your culture embrace “failing fast”?
To have a culture of quality, leaders must create an environment where failure is accepted and even celebrated as a learning opportunity. This requires an environment where people are empowered to take ownership of their mistakes, fix them, and ensure they don’t happen again. Everyone is accountable for quality.
Are you empowering QA analysts?
Note that I don’t call them “testers.” A quality analyst or strategist is a person who can analyze what’s wrong with a system and make recommendations to improve that system throughout the entire product development lifecycle. A tester, on the other hand, is reactive, offering binary pass/fail feedback that doesn’t consider methods of improvement. In this environment, QA analysts can help developers understand how they’re testing and why. Developers, in turn, can begin to write code that considers how a test will be run, thereby accelerating delivery and value.

A CI/CD pipeline produces feedback almost instantly, which offers tremendous potential in accelerating development. However, the success of the system is dependent upon leaders that champion a culture that embraces the free flow of information, tolerance of failure, and a focus on value.

If you build it (right), speed will come

Deming’s work in optimizing auto manufacturing gives us a historical context on how to build quality software products or services. Even 70 years later, his ideas are as relevant as ever. Still, software services and solutions providers, as well as IT enterprises, have been slow to adopt this second wave of Deming-inspired philosophy.

If speed-to-market is your only goal, it’s easy to slip into the quantity mindset that caused U.S. auto manufacturers to fall behind. A better approach is to create improved systems of quality — a well-oiled continuous delivery pipeline. Build it right. Speed will follow

Editor’s note: The article was first published on February 7, 2019, and was updated for relevancy and accuracy on April 21, 2021.