While I was writing a piece on newly created Dutch testing approach, I took a closer look at a couple of models in testing. In particular I tried to assess the Tester Freedom Scale by Jonathan Bach and the Heuristic Test Strategy Model by James Bach. To me, both these models are descriptive theories, which means that they try to capture and explain some of the phenomena in software testing.
In the case of the newly created Dutch testing approach, Jonathan Bach’s descriptive theory was modified into a normative theory. A normative theory is a theory that states: “If the situation is such and such, THOU SHALT do this and that (in order to achieve a satisfactory result).” I do not know if it is possible to turn a descriptive theory into a normative theory, but I think it is a dramatic switch, which, in case it happens, should probably be accompanied by a thorough scientific investigation. Such an investigation will probably take into account the data on which the original theory was based, and it will also explain why the proposed directives are the best solution in all the situations that are captured in the descriptive theory.
The fact that a normative theory – perhaps even more than a descriptive theory – should be tested thoroughly in order to produce valid instructions, is probably the reason why most of the normative theories we know in testing are watered down to ‘best practices’ in real life. If we take, for example, the theory of TMap – the most prominent normative theory of testing in the Netherlands – we see that in practice, its instructions are weakened to a toolbox of practices that are applied at the best judgement of the tester. We have come to learn that the instructions that are written in the book had better be interpreted, adapted and reshaped in order for them to make some sense.
While this ‘weakening down’ of the normative theory is hailed as an innovation by the proponents of the methodology, it signifies, in fact, that the proposed theory of testing has been invalidated. It means that “Things fall apart; / the centre cannot hold“. At the core of the theory there is a definition of testing that will probably not stand thorough scientific investigation, if ever we should come to test the theories that form our craft. An example of such a definition can be found in the normative theory created by the International Software Testing Qualifications Board, as displayed below.
The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.
Two things are striking enough to mention. The first one is that we do not consciously and willingly test our theories, nor do we invite rigorous testing. We are unable to provide evidence for, or challenge, the fundamental0 theoretical assumptions by which there is proposed one right way to approach software testing.
The second observation that intrigues me is that software testing produces normative theories at all. Yet here we are wrestling with overarching methodogies, instead of focusing on the practice and ways to describe what actually happens. If we feel the need for theories of testing, we should be focusing on descriptive theories. In an article on his weblog, Markus Gaertner expresses exactly this, in the following words.
Anyway, I think we should stop teaching testers practices that they might or might not need. Instead we should focus more on teaching testers how to evaluate their situation, and make useful improvements to their work where it’s appropriate.
I wholeheartedly welcome Gaertner’s sentiments. They parallel Philip Johnson-Laird’s handling of theories on mental modelling, to which he devotes a number of chapters of his book Mental Models. Johnson-Laird tests the theories one by one on a couple of criteria, the most important of which is that the theories have to account for the phenomena that are observed. Which brings me back to the Bach brothers and their descriptive models. I hope they will be tested rigorously.