It’s better to talk about ‘good practice’ rather than ‘best practice’ as the appropriate software testing approach is so dependent on organisational and project context. When determining an appropriate test approach we need a good understanding of the frequently conflicting priorities between product quality and the project constraints relating to timescale and budget.
The term testing best practice has a connotation of there being a single or a small number of right ways to approach software testing. This is problematic as the appropriate approach depends so much on deciding what is ‘fit for purpose’ in terms of product quality within the context of an organisation’s or a software development project’s timescale and cost constraints. It could be argued that the ISQTB® glossary of terms’ definition of best practice (2004-2014) takes context into account: “A superior method or innovative practice that contributes to the improved performance of an organization under given context, usually recognized as ‘best’ by other peer organizations” (www.istqb.org/downloads.html). Given each project’s context is unique does this interpretation mean there are potentially an infinite number of best practices? If so, talking about best practice is not particularly helpful. I like the 2011 ISTQB Expert Level syllabus’ observation when discussing models for process improvement that perhaps good practice is a better term to use rather than best practice.
Whether to position testing practices as good practices or best practices is important. I agree with Cem Kaner, James Bach and Brett Pettichord on page 261 of ‘Lessons learned in Software Testing’, Wiley 2002. Their first two of seven basic principles of the ‘Context-Driven School’ of testing are: “the value of any practice depends on its context; there are good practices in context, but there are no best practices” (www.context-driven-testing.com).
A powerful way of considering a project’s context is to represent the features being delivered by a project as a triangle. In a perfect world this would be an equilateral triangle where one side representing the need for quality would be the same length as each of the other two sides representing the project constraints relating to timescale and budget. From a Test Manager’s perspective the quality line represents the amount of software testing that can be undertaken to measure software quality. However, it’s rare for these three often conflicting priorities to be in harmony. A good Test Manager will often have two of these priorities under control, but have to devote considerable effort to dealing with the third.
The triangle is a good way of representing the impact of the three priorities on each other. For example, timescale constraints have often been severe on large projects I have worked on in the financial services sector. Time-to-market considerations were critical in terms of launching new products before competitors launched theirs. Either budget constraints had to be relaxed and/or more risk had to be taken with the quality of the product being delivered. From a testing perspective taking more risk with product quality could mean reducing the amount of testing in what were considered to be lower product risk areas. These projects were typically using Waterfall of V-model software development approaches. As the timescale constraint line increased, one or both of the budget constraint and quality lines had to been shortened in order for the project to implement the same number of features.
Is the financial services triangle example above similar for agile developments? It would be wrong to say projects using an agile approach to deliver increments within very short timescales are likely to have to compromise on quality. They would not need to shorten the quality line relative to the timescale constraint line. An agile approach could preserve the quality of the features delivered by reducing the size of the triangle - the features - to a manageable size.
The triangle analogy is a great way of considering project context, and enabling project stakeholders to reach a consensus about the optimum software development and testing approach. When a stakeholder is trying to understand how much software testing is appropriate, a nice way of positioning this is that it depends on the product risks that testing needs to help mitigate and the project constraints relating to timescale and budget. When agreeing exit criteria for the completion of each test level these very much relate to the three sides of the triangle: time, budget, defects and test coverage (the last two being represented by the quality side).
Risks and constraints can change at any time during a project. Therefore, in addition to understanding a project’s context during test planning, Test Managers also need to be considering these conflicting priorities during the monitoring and control of subsequent activities. Material changes are likely to put agreed exit criteria at risk and require corrective actions to be taken.
I have heard many different terms for this triangle analogy over the years. The best term was used by Ruud Teunissen at a BCS SIGiST conference (www.bcs.org/category/9264): ‘The Devil’s Triangle’.
27th June 2014