The theme of the seventh annual peer conference of the Dutch Exploratory Workshop on Testing (DEWT7) is lessons learned in software testing. In the light of that theme I want to share a lesson recently learned.
Broadly stated, the lesson learned is that nearly any effort in software testing develops in a non-linear way. This may seem like a wide open door, but I find that it contrasts with the way software testing is portrayed in many presentations, books and articles. It is likely that due to the limitations of the medium, decisions must be made to focus on some key areas and leave out seemingly trivial details. When describing or explaining testing to other people, we may be inclined to create coherent narratives in which a theme is gradually developed, following logical steps.
Over the last couple of months I came to realize something that I’ve been experiencing for a longer time; the reality of testing is not a coherent narrative. Rather; it is a series of insights based on a mixture of (intellectual) effort and will, craftsmanship, conflicts and emotions, personality and personal interests and, last but not certainly least, circumstance, among which chance and serendipity. The study aimed at the core of testing is the study of the decision making process that the software tester goes through.
My particular experience is one of balancing many aspects of the software development process in order to move towards a better view of the quality of the software. I spent six full weeks refactoring an old (semi) automated regression test suite in order to be able to produce test data in a more consistent manner. As expected, there was not enough time to complete this refactoring. Other priorities were pressing, so I got involved in the team’s effort to build a web service and assist in setting up unit testing. My personal interest in setting up unit testing evolved out of my conviction that the distribution of automated tests as shown in Cohn’s Test Automation Pyramid is basically a sound one. The drive to make more of unit testing was further fueled by a presentation by J.B. Rainsberger (Integrated Tests Are A Scam). I used unit testing to stimulate the team’s thinking about coverage. I was willing to follow through on setting up a crisp and sound automation strategy, but having set some wheels in motion I had to catch up with the business domain. With four developers in the team mainly focusing on code, I felt (was made to feel) that my added value to the team was in learning as much as needed about why we were building the software product. To look outward instead of inward. And this is were I am at the moment, employing heuristics such as FEW HICCUPS and CRUSSPIC STMPL (PDF) to investigate the context. It turns out that my investment in the old automated regression test suite to churn out production-like data is now starting to prove its worth. Luck or foresight?
All this time a test strategy (a single one?) is under development. Actually, there have been long (and I mean long) discussions about the test approach within the team. I could have ‘mandated’ a testing strategy from my position as being the person in the team who has the most experience in testing. Instead I decided to provide a little guidance here and there but to keep away from a formal plan. Currently the test strategy is evolving ‘by example’, which I believe is the most efficient way and also the way that keeps everyone involved and invested.
The evolution of the understanding of the quality of the software product is not a straight path. Be skeptical of anything or anyone telling you that testing is a series of more or less formalized steps leading to a more or less fixed outcome. Consider that the evolution of the understanding of quality is impacted by many factors.
Patterns of Proof 
Hi, Joris…
Thank you for a very thoughtful post, including a nice experience report.
I’d like to offer a response to a key point:
Broadly stated, the lesson learned is that nearly any effort in software testing develops in a non-linear way. This may seem like a wide open door, but I find that it contrasts with the way software testing is portrayed in many presentations, books and articles. It is likely that due to the limitations of the medium, decisions must be made to focus on some key areas and leave out seemingly trivial details. When describing or explaining testing to other people, we may be inclined to create coherent narratives in which a theme is gradually developed, following logical steps.
Over the last couple of months I came to realize something that I’ve been experiencing for a longer time; the reality of testing is not a coherent narrative.
I agree that the models that frame software development in a linear way have a poor correspondence with the reality: as one of the CDT principles states, “Projects unfold over time in ways that are often not predictable.”
I’d say, though, that one of the purposes of testing is to take something that’s confusing and complex and dynamic and variable, and to organize a story of what’s happening and what has happened. That is, we’re trying to make sense of what’s going on in the project and the product, and frame what people know, what they don’t know yet, and what they need to know in a way that they can understand and make sense of it themselves.
The evolution of the understanding of the quality of the software product is not a straight path.
I agree! It’s important for people to notice that. The testing story—at least three threads consisting of a story of the product, a story of the testing, and a story of the quality of testing—may not be linear, but I believe it can and must be coherent.
Cheers,
—Michael B.
Hello Michael,
First of all thank you for your comment. I think I did not explain well enough what I meant by coherence. You rightly state that the tester should be able to tell a coherent story about the testing effort, whenever asked, by stakeholders, to provide evidence that can be used in a decision about the development of the software.
By coherence I meant that in presentations or articles about software testing we sometimes take for granted that the road we travel in order to build that evidence is not straightforward. We sometimes present outcomes, practices or methods without the circumstances, convictions and interests that drove us to use those methods. There are, for example, numerous presentations about the ‘art’ of asking questions that talk about the sort of questions one could ask and the way they could be asked. Such presentations present some sort of framework or catalog for questioning and assume that there is a coherent structure in the application (usage) of questions in testing.
But from the past week alone, I can provide you with enough examples to show that there are so many aspects to my particular usage of questions that it would be very tricky to define my usage in the form of “if this, then that”. Alternatively, I could provide you with an overview of the types of questions I ask; why, what, where, when… etc… but that would be just another catalog of questions. Like this: I ask questions to make people laugh, I ask questions to be cynical. I ask questions to really get to the bottom of things or I use them to hide my insecurity or lack of knowledge. I ask questions to make people think about something a little bit more themselves, I ask questions to make a statement. I ask questions to show that I care about something deeply, or I ask questions to check whether someone is paying attention. That catalog is meaningless.
Mind you, I’m not saying that I am an expert questioner, just that I am aware (some of the time) of my usage of the instrument of questioning. I am mainly interested in the awareness, even in the fact that sometimes we are and sometimes we are not aware of our intentions. These choices that we make to use the instrument in a certain way and that ultimately lead us to better insight in the quality of the software product, are the examples that can lead to mastery. But I think it is hard work to gather and analyze these examples. The one thing I think we shouldn’t do is to hide the details behind the illusion of coherence: a catalog, a practice, a method. I think we are entitled to the details that drive our search for information.
I hope I made sense somehow. If not, please do tell me.
Kind regards, Joris
Pingback: Five Blogs – 2 December 2016 – 5blogs
Pingback: Giving the good advice | Patterns of Proof