Software Testing: Master Chaos With Quality Exit Criteria

For some IT leaders, software development projects can feel like an endless cacophony of defect-related rework and delays—especially for people involved In software testing and QA.  But as frustration mounts, pinpointing a path to improvement can prove difficult.  Our solution?  Start with quality exit criteria.

If you’re familiar with software testing and QA in agile software development projects, you’ve probably stumbled upon something of a universal truth: chaos is a cruel mistress. 

Unfortunately, that’s something most of us are all too familiar with.  In fact, of all major software projects, only a mere 6 percent release on time, on budget, and within their anticipated scope. 

For the other 94 percent of projects, chaos often reigns supreme.  Despite everyone’s best intentions, rework adds up, project plans go off course, and the project inevitably goes off schedule and over budget.  As costs continue to rise, the anticipated ROI begins to erode—which is why nearly half of those besieged projects wind up cancelled altogether.

So, how do you master the chaos?  That’s easy.  You need to begin at the end.  It’s all about refining your exit criteria—or “how your software testers and QA personnel define when your project is ‘done’.”

Just think about it: where else can you affect what’s considered a finished product?

Like all things, software quality exists on a continuum.  On one end, you have “measured and managed” organizations, and on the other you have ones that are completely ad-hoc—which rely less on test plans and processes then the individual heroics of their testing team.  Being on the latter side of that spectrum doesn’t always mean low-quality, off-schedule releases, but it typically does mean one thing—that these organizations don’t know when to stop testing.

Think about that: if you don’t know when to stop, it’s because you don’t know how much you’ve already done—or how much is left to go.  You could have just barely scratched the surface and wouldn’t even know it!

This issue is more commonplace than you think, as well.  Sure, an ad-hoc organization will have similarly ad-hoc exit criteria (generally an incomplete regression suite is all that’s executed, as the deadline typically drives the release date), but more sophisticated teams experience this as well.  Admittedly, their exit criteria are better (typically requiring the completion of a scoped test plan, along with the condition that all “showstopping” bugs have been remediated), but in an age where users are increasingly picky (and have more options than ever before), it’s an unsustainable status quo—especially in the era of continuous integration/continuous delivery.

At Lighthouse, we’ve long been passionate about our exit criteria.  That’s why we built our True North Testing Methodology from the ground-up so that we can test plan better than anyone else out there.  Using our proprietary code analyzer and defect prediction algorithms, we don’t just know the scope of our testing projects—we know exactly how many defects we expect to find! 

But that’s just the half of it.  No, to reach our exit criteria, our testers have to make sure that their defect reporting rate is also within its target (if they’ve reached their target, but they’re still finding defects every 15 minutes, our data says there are still a lot more bugs lurking within the code).  Think of it as a two-step verification that ensures a measurable, consistent endpoint that can be tailored to any organization’s preference and need.

All of this boils down to another fundamental truth: chaos can be mastered, but it takes a wealth of experience and dogged determination to do so.  If you’re struggling with chaos, drop us a line sometime.  We’d love to help you master it.

Cheers,

Mike Hodge
Lighthouse Technologies, Inc.
Software Testing | Quality Assurance Consulting | Oracle EBS Consulting