What the automotive industry can teach us about QA
Among software services and solutions providers, “industry-leading” speed-to-market is a commonly parroted value proposition. The underlying promise of speed is to offer business value, faster. But as we’ve discussed in past articles, if you’re not building something that produces value, how fast you’re building it is irrelevant.
To illustrate this point, we need look no further than to an industry that knows a lot about speed: Automobile manufacturing. By the late 1970s, the once-mighty U.S. auto industry was faced with a choice: evolve or die. The ideas of famed management consultant W. Edwards Deming had helped elevate a once-inferior Japanese auto industry—in particular, Toyota Motor Corporation—into an industry disruptor.
Deming’s philosophy was simple: Continuous, incremental improvements result in a quality product. By incorporating quality control principles throughout the production process, Deming contended, companies could produce the highest quality at the lowest cost. At the beginning of a packaged goods era where the focus shifted to quantity over quality, U.S. business leaders initially rebuffed the idea: Rigorous quality control was overkill and slowed production time.
These debates and ideas are as relevant as ever, particularly in software development. Deming’s focus on quality offers many insights on how we can build better software products that provide value, faster. What follows is what we can learn about software delivery—specifically, quality control—from Deming’s prescient ideas that helped transform the manufacturing industry.
Admittedly, this isn’t the first time Deming’s name has been referenced in software development circles. Deming’s work helped pioneer lean manufacturing, the principles that guided the beginnings of agile software methodology. Revisiting these ideas, however, offers great insight into how companies can improve their software development strategy.
Who Cares About Quality, Anyway?
Before diving into Deming’s theories, it’s important to discuss why having a quality product is important in the first place. We live in a time of rising customer expectations and declining brand loyalty. If a product doesn’t meet a customer’s quality standards, people move on to one that does. Even if the customer sticks around, low-quality products are burdened with an increased cost to fix and maintain.
High-quality products don’t just happen magically. They are the result of rigorous testing, feedback, and corresponding adjustments throughout the product life-cycle. To quote Deming: “Quality comes not from inspection, but from the improvement of the production process.”
Deming, Datsuns, and DevOps
We’ve established that building a quality product, no matter the industry, is the key to both long-term growth and increased productivity. What elevated Japanese auto manufacturers from being synonymous with poor-quality Datsuns (see the 25-horsepower Datsun 113 below) to gold-safety-standard Toyota Camrys was continuous improvement—small, incremental improvements to design and process over time.
In software, quality control practices occur through quality assurance (QA) and testing. Until recently, the dominant workflow for software development was the waterfall model, wherein development follows a linear path: design, code, test. In the waterfall process, each phase in the path can take months, creating an enormous gap between building and testing.
This approach is similar to those adopted by major American automakers—one that Deming criticized: Testing occurs after the product is built, meaning that any flaws, bugs, errors, or incorrect assumptions aren’t discovered until the final stages of production. This makes problems more difficult to detect and fix and can severely slow (or even halt) production.
Agile methodologies tighten design-code-test development cycles, usually into two-week sprints. This narrows the testing gap seen in waterfall, but a gap of two weeks between building and testing is still too long. To fully incorporate quality control into the production process, testing and QA must occur at each stage in the process.
This is best achieved by breaking down walls between departments and creating an environment where testing and coding information is freely shared among the team. In software development, it’s a feedback loop referred to as DevOps, the fusing of development and operations.
Beyond Agile: DevOps Adoption
DevOps and agile are not competing ideas, but rather complimentary ones. The agile manifesto lays out practices that enable cross-functional teams to iteratively deliver working software to users. All assumptions are validated by user testing in an environment that embraces flexibility rather than a predetermined roadmap.
DevOps takes this approach one step further, creating a team structure that eliminates the barriers between departments. In DevOps organizations, teams operate as a cohesive unit rather than disparate silos. Developers, testers, administrators, and even users work as a unified team that shares responsibility for building, testing, and deploying products faster.
This team structure is straight out of the Deming playbook. At Toyota, each member of the team was responsible for the quality and outcome of the final product, which created a sense of ownership and pride in the car that rolled off the assembly line.
Since we’re throwing around several buzz-wordy terms, let’s briefly define what each methodology is and where each fits within the larger picture:
- Agile methodology is the set of optimal processes that guide the day-to-day activities of a team
- Value-driven development is the compass that should guide what teams build next and why
- DevOps is the culture and organizational structure that creates the optimal team to deliver on that defined value
With each of the above practices in place, the next goal is the creation of a “continuous” pipeline:
- Continuous integration (CI): Code is continuously written into a shared repository
- Continuous delivery (CD): Design, code, test, and release software in the shortest possible cycles
- Continuous testing (CT): Testing occurs at each stage in the process with no downtime. If the code fails, it goes back to development; if it passes, it goes straight to deployment
Continuous, Automated Testing, and Clean Code
As we’ve established, when testing occurs at the final stage of a production cycle (whether that be in two-week or two-month iterations), testing becomes a bottleneck that results in lower quality and longer production times.
That’s where automated testing and continuous testing (CT) comes in. Building an automated testing platform requires an up-front cost that many organizations balk at. However, consider a construction analogy: If the product you’re building is a skyscraper, an automated testing platform is the scaffolding that must be built before a single brick is laid.
With that “scaffolding” in place, organizations can deploy code every 10 minutes rather than every two hours or two months. This produces feedback and results faster, making it easier to identify mistakes. Since testing can occur in real time, it allows the DevOps team to focus on developing new features and lowers the cost of testing in the long run. Over time, the process streamlines, resulting in continuous, incremental improvements that create quality and efficiency, faster.
But to view automated testing as merely an efficiency and speed play is to miss the point: Automation isn’t just about efficiency. When mundane tasks like testing checklists are automated, the team is freed up to focus on a higher level of quality in development and customized manual tests. The results of optimized, automated QA and testing influences how the development team writes new code—clean code that can easily fit into the testing structure.
If speed-to-market is your only goal, it’s easy to slip into the quantity mindset that caused U.S. auto manufacturers to fall behind. A better approach is to create improved systems of quality—a well-oiled continuous delivery pipeline. You’ll create a great product at reduced cost, faster.
But, I Have a CI/CD Pipeline. Why is Quality Still Lacking?
Many organizations have adopted DevOps and a CI/CD pipeline but have yet to realize any benefit. Again, systems and process are useless if they’re not aligned with organizational value. Consider these questions when evaluating why a CI/CD pipeline isn’t producing results:
- What are you trying to accomplish with a CI/CD system? Is the goal to simply move faster? If so, you’re not following the principles that made Deming and Toyota successful. A CI/CD system designed as a speed-to-market play will only result in poor quality, faster. It’s like speeding up a production line that only produces lemons. Fix the machine that builds the product—optimizing your CI/CD pipeline for quality—and the result will be a better product with less waste in the process.
- How are you automating to ensure quality? The goal of a CI/CD pipeline should be to expose problems as quickly as possible in the process of building quality. When built correctly, a CI/CD system won’t even allow you to check in the bad code. When an error is detected, halt production, fix the problem at its root, then move forward. In the process of fixing this underlying problem, developers will become better at making sure whatever error occurred doesn’t happen next time. Master the process first, then automate it.
- Does your culture embrace “failing fast”? To have a culture of quality, leaders must create an environment where failure is accepted and even celebrated as a learning opportunity. This requires an environment where people can make mistakes but are then empowered to take ownership of those mistakes, fix them, and ensure they don’t happen again. Everyone is accountable for quality. Fail fast and learn from the failure.
- Are you empowering QA analysts? Note that I don’t call them “testers.” A quality analyst is a person who can analyze what’s wrong with a system and make recommendations to improve that system. Their feedback is taken into consideration throughout the entire product development lifecycle rather than at the final stages of production. A tester is reactive, offering binary pass/fail feedback that doesn’t consider methods of improvement. This idea comes from the Deming playbook, where the analyst has an integral role in constantly improving the system. In this environment, QA analysts can help developers understand how they’re testing and why. Developers, in turn, can begin to write code that considers how a test will be run, thereby accelerating delivery and value.
A CI/CD pipeline produces feedback almost instantly, which offers tremendous potential in accelerating development. However, the success of the system is dependent upon leaders that champion a culture which embraces the free flow of information, tolerance of failure, and a focus on value.
If You Build It (Right), Speed Will Come
Deming’s work in optimizing auto manufacturing gives us a historical context on how to build quality software products. Almost 70 years later, his ideas are as relevant as ever. Still, software services and solutions providers, as well as IT enterprises, have been slow to adopt this second wave of Deming-inspired philosophy.
If speed-to-market is your only goal, it’s easy to slip into the quantity mindset that caused U.S. auto manufacturers to fall behind. A better approach is to create improved systems of quality—a well-oiled continuous delivery pipeline. You’ll create a great product at a reduced cost, faster.
Build it right. Speed will follow.
More on How to Accelerate in 2019
A deeper dive: For more information on how to achieve greater quality and delivery optimization, you can get in touch with one of our managed services and solutions experts for a personalized walkthrough.