A case for testing

Agile universe revolves around testing.  Developers are supposed to test, testers are supposed to test together with the developers.  Then the newly done work is supposed to be delivered to product owners and end users, who in turn get to try it, meaning, of course, to test it.

The result of this process is a high-quality product, where defects are known due to extensive testing, and features are exactly what the product owners and users have asked for and reviewed in their testing.    In fact, that’s the definition of a high-quality product – it does exactly what is expected of it, and its features are just what the doctor owner ordered, no more and no less.

However, there is a disconnect between the happy Agile universe, and the grim reality of software development industry. Testing is hard, expensive, decidedly not glamorous, and traditionally not valued very high.  For all the lip service that testing gets, there is relatively little research on the value of testing, how to do it right, how to measure the value, quality and quantity of the testing that does happen, and how much testing is enough. There is even less understanding of the importance of testing out in the field. 

Testing is a purely informational pursuit.  A test is something that’s done to learn how a product (a process, a person, etc.) behaves in a particular situation.  Modern software systems are very sensitive to lots of variables, i.e. “particular situation” is often extremely complicated to specify definitively.  Yet, this is what testing is all about – gathering information about what and how affects behavior of a system.

Regardless of how much work is done in testing, the reward and appreciation often goes not to the test and the people who performed it, but to the person or the team that changes (improves) the system behavior using the information gathered by testing. The idea that the improvement (or a bug fix) is wholly dependent on the information gathered in testing is not intuitive and usually gets lost in the shuffle.  A lot of information gathered by testing does not directly lead to a bug fix or improvement, and, thus, is not considered at all. 

It is hard to value the information, therefore, it is considered to be of low value.  It is easy to value the results of system improvements, therefore, they are valued highly.  As long as this mindset persists, the software industry will continue to under-test, and produce low-quality product that behaves unpredictably.

These thoughts are inspired by the recent news about Sony Playstation network compromised and subscribers' data possibly breached, private data of Texas employees handled unencrypted and left on publicly accessible website for months, Intuit's Quickbook service unavailable for many days earlier this spring.