Marco Valtas

How much quality is enough?

This is a tricky question, sounds like we can find a storage to pick quality amounts until we get enough. We know this is not it, still we struggle to talk about software quality in the context of its development, and the struggle leads companies to create magic quality gates to protect themselves.

A quality gate is something that somehow decides if the software is worth to go to production.¹ Test coverage is a classic number from where to build a quality gate. High quality software usually has high test coverage, but the catch is that coverage is a necessary but not sufficient condition to high quality.

“Testing shows the presence, not the absence of bugs.” Dijkstra (1969)

Coverage makes a bad gate. Just a quick math on the amount of conditions and combinations of inputs and states in a piece of software should be enough to discourage testing everything. Even a 100% coverage is a line execution metric and not all combinations were tested.

Someone somewhere in your company had an epiphany and uttered: “Software created here should have no less than 80% test coverage.” What about the 20% left untested? It can fail? And what if in this 20% there’s a crucial piece of code? You get the idea and also check this post for more discussion on test coverage. No magic creates correlation that does not exist, such incantations result in rule beating.

What’s happening is that people worried about software quality want to solve this problem exclusively in the domain of software development. Meanwhile the quality manifests itself in live working software. Without observing the software in its running environment there’s so much information we can obtain.

For that you need DevOps. No, you can’t buy it or create from nothing. You need to engage in practices which will approach the development people to the operations people and from this mix DevOps happens. Unless DevOps manifests itself through collaboration of development and operations, the quality of your software will be suboptimal. Start by measuring anything and everything and cultivate a curiosity for a lot of other interesting metrics which helps operations and development to find ways to improve quality.

The gates should be self-imposed by the teams from metrics that matter to them. Not by some general rule disregarding the context in which the software will be created and run.

Returning to our first question ”How much quality is enough?” The only answer is: the most we can get in this context. There are practices which improve the quality, or better, the absence of problems in software. But there’s no magic number from which enough quality magic gates can be built.