Not a Conference on Test Strategy


A response to this blog post was written by Colin Cherry on his weblog. His article is entitled (In Response to DEWT5) – What Has a Test Strategy Ever Done for Us?

On page one, line two of my notes of the 5th peer conference of the Dutch Exploratory Workshop on Testing — the theme was test strategy — the following is noted:

Test (strategy) is dead!

And scribbled in the sideline:

Among a conference of 24 professionals there seems to be no agreement at all on what test strategy is.

In putting together a talk for DEWT5 I struggled to find examples of me creating and handling a test strategy. In retrospect, perhaps this struggle was not as much caused by a lack of strategizing on my part, as it was caused by my inability to recognize a test strategy as such.

Still I find it utterly fascinating that in the field of study that we call ‘software testing’ — which has been in existince since (roughly) the 1960s — we are at a total loss when we try to define even the most basic terms of our craft. During the conference it turned out that there are many ways to think of a strategy. During the open season after the first talk, by the very brave Marjana Shammi, a discussion between the delegates turned into an attempt to come to a common understanding of the concept of test strategy. Luckily this attempt was nipped in the bud by DEWT5 organizers Ruud Cox and Philip Hoeben.

For the rest of the conference we decided to put aside the nagging question of what me mean when we call something a test strategy, and just take the experience reports at face value. In hindsight, I think this was a heroic decision, and it proved to be right because the conference blossomed with colourful takes on strategy. Particularly Richard Bradshaw‘s persistent refusal to call his way of working — presented during his experience report —  a ‘strategy’, now does not stand out so much as an act of defiance, yet as an act of sensibility.

A definition of test strategy that reflects Richard’s point of view and was mentioned in other experience reports as well,  is that a strategy is “the things (that shape what) I do”.

And yet I couldn’t help myself by overturning the stone yet one more time during lunch on Sunday with Joep Schuurkes and Maaret Pyhäjärvi. Why is it that we are in a field of study that is apparently in such a mess that even seasoned professionals among themselves are unable to find agreement on definitions and terms. I proposed that, for example, the field of surgery will have very specific and exact definitions of, for example, the way to cut through human tissue. Why don’t we have such a common language?

Maaret offered as an answer that there may have been a time in our field of study when the words ‘test strategy’ meant the same thing to a relatively large number of people. At least we have books that testify of a test strategy in a confident and detailed way. The fact that the participants of the fifth conference of the Dutch Exploratory Workshop on Testing in 2015 are unable to describe ‘strategy’ in a common way, perhaps reflects the development of the craft since then.

Tower of Babel, Pieter Bruegel

The Tower of Babel by Pieter Bruegel the Elder (1563)

As a personal thought I would like to add to this that we should not necessarily think of our craft as a thing that progresses (constantly). It goes through upheavals that are powerful enough to destroy it, or to change it utterly. It may turn out that DEWT5 happened in the middle of one of these upheavals; one that forced us to rethink the existence of a common language. The biblical tale of the tower of Babel suggests that without a common language, humans are unable to work together and build greater things. Perhaps the challenge of working together and sharing knowledge without having access to a common language is what context-driven testing is trying to solve by adhering to experience reports. ISTQB and ISO 29119 are trying to fix the very same problem by declaring the language and forcing it upon the testing community. This is a blunt, political move, but, like the reaction from the context-driven community, it is also an attempt to survive.

With regards to my ‘surgery’ analogy, Joep suggested that surgeons deal with physical things and as such, they have the possibility to offer a physical representation of the definition. Software testing deals with the intangible, and as such our definitions are, forever, abstractions. If we want to look for analogies in other domains then perhaps the field of philosophy is closer to software testing. And in philosophy the struggle with definitions is never ending; it runs through the heart of this field. Maybe it is something we just need to accept.

The Cheeseburger Standard


Last evening I picked up Peopleware by Tom DeMarco and Timothy Lister. It has been, for quite some time now, a book that I open when I want an entertaining view on software development that stipulates some home truths about the game. I opened up chapter 2, read a couple of pages and was struck by how much this text relates to the ISO 29119 debate.

Chapter 2 – entitled ‘Make a cheeseburger, sell a cheeseburger’ – starts off as follows (in all quotations, emphasis mine).

Development is inherently different from production.

I am still, and always will be, greatly in awe of the fact that quite a number of people in the software industry are blissfully unaware of this and go full Taylor on everything that moves.

But managers of development and allied efforts often allow their thinking to be shaped by a management philosophy derived entirely from a production environment. Imagine for the moment that you’re the manager of the local fast food franchise. It makes perfect sense for you to take any or all of the following efficient production measures:

  • Squeeze out error. Make the machine (the human machine) run as smoothly as possible.
  • Take a hard line about people goofing off on the job.
  • Treat workers as interchangeable pieces of the machine.
  • Optimize the steady state. (Don’t even think about how the operation got up to speed, or what it would take to close it down.)
  • Standardize procedure. Do everything by the book.
  • Eliminate experimentation – that’s what the folks at the head-quarters are paid for.

These would be reasonable approaches if you were in the fast food business (or any production enironment), but you’re not. The “make a cheeseburger, sell a cheeseburger”mentality can be fatal in your development area. It can only serve to damp your people’s spirit and focus their attention away from the real problems at hand. This style of management will be directly at odds with the work.

To manage thinking workers effectively, you need to take measures nearly opposite those listed above.

And further on, on the subject of making errors.

Fostering an atmosphere that doesn’t allow for error simply makes people defensive. They don’t try things that may turn out badly. You encourage this defensiveness when you try to systematize the process, when you impose rigid methodologies so that staff members are not allowed to make any of the key strategic decisions lest they make them incorrectly. The average level of technology may be modestly improved by any steps you take to inhibit error. The team sociology, however, can suffer grievously.

Further on, on the subject of the steady state.

Steady-state production thinking is particularly ill-suited to project work. We tend to forget that a project’s entire purpose in life is to put itself out of business. The only steady state in the life of a project is rigor mortis. Unless you’re riding herd on a canceled or about-to-be-canceled project, the entire focus of project management ought to be the dynamics of the development effort. Yet the way we assess people’s value to a new project is often based on their steady-state characteristics: how much code they can write or how much documentation they can produce. We pay far to little attention to how well each of them fits into the effort as a whole.

And lastly, on the subject of doing without thinking.

If you are charged with getting a task done, what proportion of your time ought to be dedicated to actually doing the task? Not one hundred percent. There ought to be some provision for brainstorming, investigation, new methods, figuring out how to avoid doing some of the subtasks, reading, training and just goofing off.

The steady-state cheeseburger mentality barely even pays lip service to the idea of thinking on the job. Its every inclination is to push the effort into one hundred percent do-mode.

The dilemma in software testing that is characterized by ISO 29119 is whether we regard software testing as a factory process or as an act of human investigation. As the quotations from Peopleware showed, this dilemma is far from new. The ISO 29119 people may strongly doubt the experiences written down by Tom DeMarco and Timothy Lister as much as we (context-driven) testers doubt the way of working that is imposed by the Central Committee Working Group 26. I choose to believe that software testing is an act of investigation because from what I’ve experienced so far, the reality of software development looks much like it is described by DeMarco and Lister. If, however, the reality of software development is the exact opposite of what is described by DeMarco and Lister and the factory approach does indeed lead to better software each and every time, then I think the backers of the ISO 29119 standard should come forward, refute the evidence of software development as a human act, and convince us by showing us the reality of software development as they experience it.


Tom DeMarco, Timothy Lister (1999). Peopleware. Dorset House Publishing Company.