Effective test approaches, efficient tool support, testing certification

You can't put lipstick on a pig!

Pig with lipstick

Many organisations still believe that software testing is solely concerned with executing code and observing whether tests pass or fail. Running tests against poor quality software is actually a very ineffective and inefficient way of trying to deliver products that are fit for purpose.

You can't put lipstick on a pig.

It is better to test at the earliest opportunity using reviews and static analysis in order to help build quality into software development products from the outset of the development of those products, rather than waiting for code to be compiled before starting testing.

  • Barry Boehm concluded in 'Software Engineering Economics', Prentice Hall, 1981 (pp 39-41) that errors were typically one hundred times more expensive to correct in maintenance for larger projects than if they had been removed at the point they were introduced (e.g. through reviews during requirements definition). The quick correction of an ambiguous requirement statement is much more cost effective than users finding during acceptance testing or after implementation that a major feature has not been implemented as required.
  • The original ISEB/BCS Foundation Level syllabus (V2.0 – 25 February 1999) positioned this in respect of the economics of early testing: “the cost of testing is generally lower than the cost associated with major faults”.
  • The sub-title for the 6th World Congress for Software Quality is "Shift left - inspiration and innovation for software quality". The 'Shift left' positioning of building quality into software developments products was being written about as long ago as 2001 by Larry Smith (http://www.drdobbs.com/shift-left-testing/184404768). The adoption of 'Shift left' by the WCSQ is indicative of the slowness of the software development industry to be able to put sufficient emphasis on building quality into software products up front.
  • It is worth remembering that Barry Boehm was making his 1981 comments in Software Engineering Economics in the context of effective use of the Waterfall model through verification ("are we building the product right") and validation ("are we building the right product", Boehm p37) whilst specifications and code were being built. The V-model though is a more effective model for aligning different levels of testing, each with their own objectives, with the main software development products. It is a powerful model for explicitly depicting the early involvement of testing in the verification and validation of specifications and code.
  • Relying on a documentation driven approach with extensive use of verification and validation can't guarantee product quality, and in many contexts isn't appropriate. Barry Boehm also concluded based on data for smaller projects (using approaches such as prototyping) that different cost-to-fix escalation curves indicated trade-off points where proceeding with a prototype could be more cost efficient than trying to pin down requirements in full detail. The 'Shift left' message still applies to iterative-incremental development models though, as evidenced by the Test Driven Development approach taken in agile methods.

Why are so many organisations still relying on running tests to find failures rather than thinking in terms of 'Shift left'?

The answer lies in related cultural attitudes which have proved to be hard to change.

  • An attitude I have often experienced is that 'Anyone can test' as testing is inherently easy: 'you just run tests against the software don't you?' I blogged about this in 2013 at NCC Group (https://www.nccgroup.com/en/blog/2013/02/anyone-can-test/).
  • This, coupled with an expectation that exhaustive tests can be run against software, has helped perpetuate the attitude that software quality can easily be retrofitted into software after it has been built. Glenford Myers illustrated the implication of trying to test exhaustively in 'The Art of Software Testing', Wiley, 1979. A relatively simple program which allowed a choice of five options, and which could be repeated up to twenty times, would have 100 trillion paths. It would take a billion effort years to undertake all path testing at five minutes per test (pp9-11). This exhaustive path testing would not have included any negative testing, and neither would operating conditions have been varied. There would have been even more testing to do to achieve exhaustive testing.
  • A significant change for the better was the publishing of the Agile Manifesto in 2001 (http://www.agilemanifesto.org). Four values particularly applicable to Waterfall and V-model developments were contrasted with four values that Agile places even more importance on. Processes, documentation, contracts and plans have value in projects. However, the Agile Manifesto places even more value on individuals and interactions, working software, customer collaboration, and responding to change. Waterfall and V-model development lead times can be so long that the risk associated with changing requirements can outweigh effective verification and validation. A high proportion of organisations are trying to shift their values leftwards towards the agile end of the spectrum for some or all of these four pairs of values.
  • Finally in addition to shifting left there needs to be as much emphasis on 'shifting up' to defect prevention through process improvement as opposed to focusing solely on defect finding. Many organisations go through the same experiences project after project in terms of finding high levels of defects in documents and code. Investing in the root cause analysis of these defects, learning lessons for future projects, and then implementing the right changes at the right time, can provide a virtuous circle in terms of improved software quality from one software development to another.

In conclusion, it is good news that 'shift left' should be given such a high profile at the 2014 World Congress for Software Quality. At the same time it's frustrating that so many organisations have not moved forward further over the last thirty years. Perhaps the difference today is that attitudes and values towards how we go about software development, software testing and process improvement are changing for the better.

A few years ago a delegate emailed the above picture of a pig to me after an ISTQB Certified Tester Foundation Level course.

My reply was that it would still be a pig no matter how much lipstick he applied to it.


23rd May 2014