Pointing to Pyramids


When we look at the collection of test automation pyramids that have been published over the last couple of years, it is hard to get a clear picture of its (original) purpose and the significance of its variations. It appears that every test automation pyramid we encounter is a new model in itself due to sometimes slight and sometimes fundamental adjustments. There is no doubt about its popularity as a model in software testing. But its development and use is troubled by the rather reckless treatment of its lineage.

There are many reasons why the field of software testing easily loses track of what has been produced in the past. One of the reasons is that it is hard to find good references. Finding references requires the study of literature and this is something that is often taken for granted. As an example of how the field of software testing obfuscates rather than clarifies the history, the evolution and the use of its models, I would like to take a few lines from an article that was recently published in the magazine Tea-time with Testers.

Up front I must mention that Tea-time with Testers is a magazine that is offered for free to the testing community. It is platform for those who desire to contribute to the field. I think we should appreciate any initiative that tries to improve the state of testing and that we should respect the people, such as Lalitkumar Bhamare (editor of Tea-time with Testers) who invest their time in these initiatives. Furthermore we should respect the writers who invest their time in sharing their experiences with the community.

In the January edition of Tea-time with Testers there is an article entitled The Agile Testing Pyramid: Not so much about Tools But more about People And Culture. In the article a reference is made to Mike Cohn’s test automation pyramid. The article itself is an experience report of the implementation of a test automation strategy in accordance with the test automation pyramid. There are some interesting conclusions and on the whole it is nice to read about the application of the pyramid in practice.

It is not my goal to criticize the article. Merely, I want to use the way it references the test automation pyramid as an example of how inaccuracies in referencing can obscure the intentions of a model. I could have taken another article or even a book about software testing to show that same lack of accuracy or sometimes even the total lack of acknowledgement of the existence of predecessor.

At the end of the article the following is stated:

The Agile Testing Pyramid is an agile test automation concept developed by Mike Cohn.


For more information: Succeeding with Agile: Software Development Using Scrum. Mike Cohn, Pearson Education, 2010

The Agile Testing Pyramid as shown in the article

The picture besides the text in the article shows the test automation pyramid with the three layers: unit, service and UI as developed by Mike Cohn. The pyramid that is shown in the article strongly resembles the pyramid that is published in the book Succeeding with Agile, which was published by Addison-Wesley in November 2009, ISBN 0321579364. It is highly likely that the (2010) reference in the article actually intends to point to this book, but we cannot be one hundred percent sure. Here is why…

The article states that Mike Cohn’s concept is called the Agile Testing Pyramid. But in the text of Succeeding with Agile, there is no mention of an Agile Testing Pyramid. Cohn calls his concept the test automation pyramid, throughout the entire book. So can we assume that there has been slight inaccuracy, that the naming of the pyramids (remember there are many of them) has gotten mixed up, but that the article still intends to point to Cohn’s 2009 pyramid? Maybe we can.

There are in fact quite a number of articles and blogs that refer to Cohn’s 2009 pyramid as the Agile Testing Pyramid, so it is not uncommon to make this mistake. There is even a book called Agile Swift by Godfrey Nolan that refers to Cohn’s 2009 pyramid as the Agile Testing Pyramid and displays an image of a totally different pyramid. Any awkwardness in this area, however, can be avoided by looking up the pyramid in Cohn’s book and referring to his work correctly.

The test automation pyramid by Mike Cohn, 2009

The question we should ask ourselves is whether test automation pyramid means something different than agile testing pyramid. It seems that the testing community believes that these names can be used interchangeably. And practitioners in the field throw other names into the mix, such as the testing pyramid, the software testing pyramid, the test pyramid, or the agile test automation pyramid. But if we do not use the name that was assigned to the model by its author, how can we be sure we indicate that particular model? We can, for example, reference the source. Without that the name Agile Testing Pyramid can mean just about anything. Luckily the article mentions the source of the pyramid. But as we saw the reference is slightly incorrect. We already suspected that the authors intended to point to the 2009 book. If there is any remaining doubt at all, this is removed by the fact that a picture of a model closely resembling Cohn’s 2009 model is shown in the article. These three facts combined, though they each have their flaws, point to the intended model. In the article that lies before us we need this three-step method for establishing that we have the correct model in mind.

  • Indicate the name of the model.
  • Make a reference to the publication of the model.
  • Add a picture of the model.

If the name and the reference would have been correct, the first two steps would have sufficed. And yet we are lucky that the authors of the article spent some effort trying to guide us to the intended model. Sometimes we are not so lucky. Sometimes Cohn’s 2009 pyramid is referred to as Cohn’s pyramid. The fact that we know of at least two disparate pyramids that were published by Mike Cohn (one in 2004 and one in 2009) means that referring to Cohn’s pyramid is not enough to point to the intended model.

Another delightful example of creating mysteries using references is the aim to point to Cohn’s 2009 pyramid by referring to it as the original pyramid. In the text of the article this phrase is used too.

The original agile testing pyramid knows three levels: unit tests, services and UI tests

The other original pyramid: the test automation pyramid by Lisa Crispin and Janet Gregory, 2009

At this point in the article we already know which model is intended by the original agile testing pyramid. Nevertheless, the phrase is wrong. If we define original as earliest then Cohn’s 2009 pyramid is not the original. There are at least a couple of test automation pyramids that were published before November 2009. None of these pyramids seem to have been popular enough to leave a lasting impression on mainstream testing. But to designate Cohn’s 2009 pyramid as the original would be distorting the truth. Even if we leave out the lesser known models there is one contender for the title of original testing pyramid, which is the model that was published by Lisa Crispin and Janet Gregory in the popular book Agile Testing in January 2009.

One of the root causes of pyramid mayhem in software testing is that there is huge amount of uncertainty about what came first and what came in what particular order. Because of this it is nearly impossible to discern any pattern or direction in the development of the test automation pyramid as a concept. If we take the Utopian view of the development of software testing we hope that it will go along the lines of the scientific method. This way a model evolves from a certain initial model and newer versions are created by testing, building on, extending and refining that initial model. In reality what we have is a Cambrian explosion of models because we do not know which models exist and how they have been tried. Each sloppy reference just stirs the soup and makes it a little bit murkier. It is because of this that the following situations may happen in practice.

  • The author is blissfully unaware of any test automation pyramid. It seems to him that his idea is new to the field of software testing. Must. Publish.
  • The author has heard about the existence of a test automation pyramid. He googles for it and after having skimmed the first three search results decides that his idea is new and fresh and the world needs to know about it.
  • The author has heard about the existence of a testing automation pyramid. He googles for it and reads the first search result. The pyramid in the first search result was published a couple of years ago and does not reference to any older pyramid. He decides that his pyramid is an improvement over the pyramid shown in the first search result and decides to publish and reference to the first search result.
  • The author has a bit more knowledge of the field of software testing. He has read the book Agile Testing by Crispin and Gregory. He knows there is a pyramid in that book. He publishes a piece about the development of the Agile pyramid and references the book as containing the original Agile pyramid.
  • The author has found Mike Cohn’s 2009 test automation pyramid. He references the pyramid in his own work and places beside it a picture of a totally different pyramid. I have actually found two examples of this, so there should be more.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s