The 5x rule and how software defects get more expensive with time

Let’s start with a disclaimer:

We talk about the 5x rule involving the costs of defects in this article, but it can get much worse than 5x. Depending on when a defect is identified, the costs associated with correction, and worse yet, the revenue losses associated with that defect, can all mean the total amount can easily surpass the 5X quote.

While we can hope for and hire the best developers in the industry, the fact remains that no one is perfect. That’s precisely why testing is a critical component in any software development project.

Let’s explore a real-life example of a defect that made its way to production.

The Samsung Note 7 lost a fortune in profits because of a bug in the battery management system.

The design: the battery management system was designed to monitor the electric current and automatically stop the charging process as soon as the battery was full.

What happened: a defect in the battery management system made its way undetected to production, much to the detriment of their customers and reputation. This bug then allowed the batteries to overcharge, sometimes leading to explosions.

The cost: reputation and liabilities aside, the cost to fix this defect was nearly $17 billion. Imagine the money, embarrassment, and headaches that might have been saved had this error been caught in the early stages of development.

Yes, this example is a bit on the extreme, but it remains very real, and a milestone to show us one of the worst-case scenarios. It is also a perfect example of how costs multiply as they go further through a process without resolution. Compare the cost of writing an extra test case to $17 billion in triaging the leaked bug, adding additional QA support makes a lot of sense.

Core things to consider when fixing defects early

Less mess

Think of an upside-down pyramid. All of the pieces at the top are dependent on the unstable blocks on the bottom. If you think of the bottom of the pyramid as user stories or requirements and the next layer as architecture and design, then as coding continues and builds off of the flawed bottom layers (represented as the unstable bottom blocks), the dependencies of that undetected instability keep growing. If those bottom layers of the pyramid are defective, those perfectly-balanced dependencies come toppling down. And, they can come crashing down hard!  The final result is a pile of rework to correct not only the defective architecture and code, but also checking, recoding, and retesting each and every dependent function to that code. Imagine for a second how much work and backtracking such a scenario could impact your project.   At Lighthouse, we call this “unplanned rework” because it’s not built into your program plan and it will likely require numerous sprints to clean it all up.

Faster recollection

When defects are detected early, as in as fast as the code is written (or very shortly after), it is still fresh in the developer’s mind. They can find the flaw quickly, make a fast correction, and move on with minimal cost. On the other hand, as time passes and the code remains until later stages and sprints, developers must take additional time to hunt down the defective code, make corrections, and they may even need to relearn the feature in order not to inject new bugs into the system.

Pro Tip: Automation testing is the solution to immediately test code as fast as that code is written.

Pro Tip #2: Defects in user stories can wreak havoc across an organization.  If they go undetected, they impact the architecture, design, and code.  Be sure you have a well-structured user story inspection process as part of your grooming activities.

Quick time to market

The sooner we identify bugs, the faster they are corrected, which leads to a faster time to production. More code into production, quicker means time to market is reduced, and faster time to market means revenue sooner. Combine that with the notion that most well-tested systems also have less associated technical debt and you’ll find your profits trending in the right direction if you identify bugs early.

The public eye

The last and probably worst outcome is if a defect manages to leak into production. It’s bad enough if you have a flaw in an organizational (internal) application, but it’s even worse if that flaw shows itself to the public.

Recall the previous example from Samsung. They managed to fix the bug eventually, but it’ll take years to repair the brand’s damaged reputation. Once the software is released and out in the wild being used and depended upon, it’s not only extra-challenging to locate defects, but the risk also increases. Additionally, production code quickly becomes business-critical from an operational perspective. The details around finding a defect and making an emergency change can quickly raise the cost as much as 30X compared to what it might cost if detected and corrected early.

Process makes a difference

Having a refined process and a well-structured SDLC will dramatically improve the success of testing efforts, so when QA and a quality program are layered onto the mature SDLC, the sky is the limit for performance and quality. The significant change here is that by having the process act as a control mechanism, the tools and techniques QA analysts and developers use to ensure quality are enabled to be as effective as possible.

Pro Tip: We do recommend an agile approach, if at all possible. In an Agile project framework, feedback is given consistently during each sprint’s retrospective. It provides a formal mechanism every couple of weeks to learn and improve your processes.  This practice keeps development costs lower and predictable through the majority of the project, and there’s less risk of cost spikes at the end of a project due to rework.

How’s your efficiency?

Efficiency isn’t as well-tracked on most teams as they think. One interesting detail is that even many Agile teams fail to track and compare how much time they spend on each sprint developing new functionality vs. correcting defects from previous sprints.

What we observed is as they continue sprinting, teams tend to spend more time fixing bugs and refactoring than they do developing new functionality. When we introduced a method of tracking these details, it became very enlightening as an indication of the quality of their work. This KPI is commonly called Defect Removal Efficiency (DRE), and it’s a means to illustrate the quality in each release. Just like software testing, you cannot correct what you don’t measure!

Partner with the right people

Lighthouse Technologies brings our clients’ QA to new levels, not only through our True North Testing Methodology™, which provides a structured approach to putting quality into your software systems but also by introducing new ways to increase quality and efficiency throughout your entire SDLC. Contact us to learn how we can not only assist in bringing your project in on-time and on-budget but also how you can make significant improvements with your software development operations.