Giving the good advice

Standard

In my previous post I tried to explain that software testing, as it happens in practice, cannot be represented as a logical and orderly (coherent) sequence of events. There are too many factors—specific circumstances— that influence the decision making process that guides testing. The study of testing is the study of this decision making process. It is driven by detailed information about the circumstances. And yet, in many publications about software testing, these details are disregarded. What remains is a representation of testing that is orderly, classified and coherent; a representation that is often lacking details (facts) by which we can verify the selected approach, method or model.

One such representation—which I mentioned in my comment on the previous post—is the catalog; a list of systematically arranged items. Usually, such a list contains items that are abstracted from reality. Items can be resources (such as software testing tools), skills, characteristics, methods, practices, preferences and many other things. The list can have many purposes; it can serve, for example, as a checklist, an overview, a heuristic, a process description or a guideline. In software testing we have a wide diversity of catalogs and my argument is that we focus too often on achieving an orderly, classified and coherent representation, while forgetting about reality.

As an example I would like to discuss a catalog that is presented in the EuroSTAR blog post G(r)ood testing 25: Tips for how to boost Unit testing as a Functional Tester. The catalog that is offered is called ‘Tips for boosting Unit testing’. In order to understand what the list looks like, a part of it is presented below.

Tips for boosting Unit testing

  • Focus on doing useful tests (tests deliver information) rather than discussing whether the tests are unit tests or functional tests
  • Ask the developer for Help (or offer him to help?), rather than telling him he needs to do UT
  • Pair the tester and developer together while defining the tests
  • Demonstrate the damage of not doing UT – E.g. showing that late found defects slow everyone down
  • Organize coding Dojo’s where you put unit testing on the agenda
  • Introduce test design techniques to the developers
  • Get an understanding about the build and deployment process

I am currently involved in setting up unit testing in an Agile team, so the topic of unit testing—and putting unit testing on the agenda—is familiar to me. When I read through the list that is mentioned in the blog post, at first glance I feel that the advice that is presented is sensible and practical. I even applied some of the tips that are on the list. It appears to contain solid and practical advice to someone wanting to make more of unit testing.

But is it really solid and practical advice? The fact that it looks very familiar to me, might suggest that it is folklore, which is defined by Merriam-Webster as “an often unsupported notion, story, or saying that is widely circulated.” Could it be that these tips are widely circulated and therefore look very familiar to me? This could be an interesting area for further investigation. And then the next question: is it good advice? Some state that “most advice is terrible”. This statement is made in one of the first links that turned up when I searched for the phrase “good or bad advice” in Google. It is interesting, because it suggests that this list of tips for unit testing that we a looking at is likely to be terrible. But it also suggests that we probably should look further and come to a more nuanced perspective of what advice means. In order to get to this nuanced perspective we need to study the nature of the advice, its context, the persons involved etc…

In order to evaluate the advice that is given to us, we need to know more about it. The blog post reveals quite a number things. The list of tips was the outcome of a brainstorm session that was held during the the 21st testing retreat in Château de Labusquière à Montadet in France. We learn that senior testers (twelve in total) from various countries were present. The fact that all testers were senior suggests that the advice has been derived from years of experience in software testing. With the means at hand it is not possible to find out if that experience encompasses unit testing, so we have to assume this. But all in all, the word ‘senior’ suggests that the advice should be good, since it must have been tested in practice extensively.

There are other aspects of the advice that can be learned from the text. For example, a motivation for giving the advice is stated (to help testers in their struggle) and in a very general sense a couple of situations are described in which the advice might be of use. But apart from the fact that the advice is likely to come from experience, we have no other indications that the advice is good. Moreover, we lack information by which we can verify whether the advice is good or not. Sure, one can apply each of the tips to the best of one’s abilities, but if this leads to (horrible) failure, how does this reflect on the quality of the advice? Perhaps the advice was bad indeed but the failure to deliver may also have been caused by shortcomings on the tester’s part. Perhaps she did not understand the advice or she lacked of skills to apply it. Furthermore there may have been circumstances adversarial to implementing the advice. Perhaps the timing was wrong, the order in which it was applied was incorrect or certain preconditions were not met.

By now it should be obvious that we lack two things. Firstly, the criteria by which we can evaluate the success or failure of the advice that is given are not present. And secondly we lack information about the context in which the advice is applicable and what it needed to apply it. Without these criteria it is possible to give any sort of advice. If I wanted for example to get from Utrecht in the Netherlands to New York, the advice might be to get on a plane. This advice contains a huge amount of implicit assumptions that are essential to the success of the advice. In other words; the advice is useless. Perhaps the same can, for example, be said of the advice to “introduce test design techniques to the developers.” I can see how test design techniques help us make an informed decision about domain coverage, but aren’t test design techniques usually based on detailed functional specifications? What is there to cover when we are coding and learning about the functioning of the application in parallel? And how do we know what technique to apply if we know little about risk? And if a test design technique tells us that we should write say thirty unit tests, how would I deal with the boredom of writing those tests? How would I handle the frustration of the developer who wrote these elaborate design-technique-based tests and has to throw them out three sprints later because of new insights with regards to the functionality? And which tests should I write that are not based on test design techniques? And what should they be based on?

As an afterthought, it may be a nice exercise in critical evaluation to add tips to the list and see if the list gets better or deteriorates because of it. If creativity is needed just look up the Celestial Emporium of Benevolent Knowledge, which is a brightly shining example of a catalog.

Advertisements

Never in a straight line

Standard

The theme of the seventh annual peer conference of the Dutch Exploratory Workshop on Testing (DEWT7) is lessons learned in software testing. In the light of that theme I want to share a lesson recently learned.

Broadly stated, the lesson learned is that nearly any effort in software testing develops in a non-linear way. This may seem like a wide open door, but I find that it contrasts with the way software testing is portrayed in many presentations, books and articles. It is likely that due to the limitations of the medium, decisions must be made to focus on some key areas and leave out seemingly trivial details. When describing or explaining testing to other people, we may be inclined to create coherent narratives in which a theme is gradually developed, following logical steps.

Over the last couple of months I came to realize something that I’ve been experiencing for a longer time; the reality of testing is not a coherent narrative. Rather; it is a series of insights based on a mixture of (intellectual) effort and will, craftsmanship, conflicts and emotions, personality and personal interests and, last but not certainly least, circumstance, among which chance and serendipity. The study aimed at the core of testing is the study of the decision making process that the software tester goes through.

My particular experience is one of balancing many aspects of the software development process in order to move towards a better view of the quality of the software. I spent six full weeks refactoring an old (semi) automated regression test suite in order to be able to produce test data in a more consistent manner. As expected, there was not enough time to complete this refactoring. Other priorities were pressing, so I got involved in the team’s effort to build a web service and assist in setting up unit testing. My personal interest in setting up unit testing evolved out of my conviction that the distribution of automated tests as shown in Cohn’s Test Automation Pyramid is basically a sound one. The drive to make more of unit testing was further fueled by a presentation by J.B. Rainsberger (Integrated Tests Are A Scam). I used unit testing to stimulate the team’s thinking about coverage. I was willing to follow through on setting up a crisp and sound automation strategy, but having set some wheels in motion I had to catch up with the business domain. With four developers in the team mainly focusing on code, I felt (was made to feel) that my added value to the team was in learning as much as needed about why we were building the software product. To look outward instead of inward. And this is were I am at the moment, employing heuristics such as FEW HICCUPS and CRUSSPIC STMPL (PDF) to investigate the context. It turns out that my investment in the old automated regression test suite to churn out production-like data is now starting to prove its worth. Luck or foresight?

All this time a test strategy (a single one?) is under development. Actually, there have been long (and I mean long) discussions about the test approach within the team. I could have ‘mandated’ a testing strategy from my position as being the person in the team who has the most experience in testing. Instead I decided to provide a little guidance here and there but to keep away from a formal plan. Currently the test strategy is evolving ‘by example’, which I believe is the most efficient way and also the way that keeps everyone involved and invested.

The evolution of the understanding of the quality of the software product is not a straight path. Be skeptical of anything or anyone telling you that testing is a series of more or less formalized steps leading to a more or less fixed outcome. Consider that the evolution of the understanding of quality is impacted by many factors.