A primitive attempt at analysis


Jari’s challenge

A couple of days ago I browsed through the puzzles (testing challenges) posted by Jari Laakso on his weblog. I picked one that I found particularly interesting and tried to come up with an answer. I sent an e-mail to Jari and he replied, thanking me for my input. I did not provide the explanation that was on his mind, but this was mainly due to the fact that I should have tried to solve the challenge in a direct discussion with him, rather than by e-mail. I intend to do this later on, but in the meantime Jari’s challenge nevertheless got me thinking about analysis; the way I tried to tackle his challenge. So thanks to Jari for starting this train of thought!

The challenge is Testing Challenge – Puzzle #3 and goes as follows.

There is a 15 year old boy studying in a high school. He loves ice hockey and is the best of the team from his year. The team has been excellent in the high school championships. Recently, the dean and the teacher’s council had a meeting where they decided he is so good they must dismiss him from the team. Explain why.

Now clearly we need some form of reasoning to find the possibilities that are hidden in this text and to account for those findings. We need our old school analytical skills; the skills that have been honed to perfection through years of intensive study, training and practice.

What the holy book has to say

One of the corner stones of our craft is the 752 page testing bible TMap Next.The book has a lot to say on analytical skills, particulary in chapter 4 Foundations of Software Testing – Applying the Analytical Mind. While this chapter dives deep into the history of analytical thinking, scientific reasoning and experimentation, it also provides what I think is the best argument for the all pervading importance of analytical skills in our craft. Chapter 4.2.1, right down the middle of page of 154 goes as follows.

To fully understand the testing experience a thorough grasp of analytics and reasoning is imperative. In the previous chapter we concluded that within any software development effort the tester is confronted with systems, models of systems, descriptions of systems or opinions on the functioning of systems that are incomplete, ambiguous, inconsistent, ill structured, contradictory, distorted or just plainly incorrect. And yet this information is seldom presented accompanied with a list of footnotes explicating in full detail its inconsistenties. Why? Because the tester is presented with information that most other people believe, at the very least partly,  to be correct, justified and validated. The task of the tester is to confront people with their beliefs, to show not the ways in which the system will succeed, but the situations in which the beliefs we hold may differ from what we encounter in real life. This is the test; and the tools we supply to the tester are the ways to reveal what is hidden. Within this set of instruments we find logic and reasoning and at the same time instruments to disrupt logic and reasoning. In the remaining parts of this chapter the most important of these instruments are discussed.

Armed with these instruments we move toward the challenge.

The challenge

There is a 15 year old boy studying in a high school. He loves ice hockey and is the best of the team from his year. The team has been excellent in the high school championships. Recently, the dean and the teacher’s council had a meeting where they decided he is so good they must dismiss him from the team. Explain why.

First there is a psyhological hurdle we must overcome. The reason for the dismissal of the boy does not seem logical; we live in a society in which being good at something is valued, praised and awarded. Being dismissed from a team at first glance hardly seems to be an award. So the challenge is to explain this ‘ unlogical’ dismissal. If the dismissal were ‘logical’ it would hardly require explanation. Since I decide that the dismissal is unlogical I have a number of assumptions I must validate. For example, I assume that the boy lives in modern times and not a couple of hundred years ago. The best clue in the text as to the time in which this challenge must be placed is the fact that ice hockey is mentioned. If we take Wikipedia as a valid source we learn that modern ice hockey started in 1875 in Montreal.

The second assumptions is that the boy lives in a modern Western culture not unlike mine. Other cultures may have totally different views of what is ‘good’ and what ‘dismissal’ means. In fact I think this challenge can only be ‘solved’ by someone from a Western culture. Luckily evidence like ‘ice hockey’, ‘high school’ point to our cosy modern Western world.

As a side note I think the information systems we create highly depend on the society in which they are implemented and on the time they are implemented. If you take this one step further you may as well state that the information systems we create are mirror images of (aspects of) the societies we live in. If you take it yet one step further you can state that information systems are elements of social systems and information systems are largely social, not technical.

Back to the psychological hurdle. The boy is dismissed because he is so good. If this is true then maybe the boy’s excellence may yet still lead to a reward, for example being promoted to a better team. On the other hand, if no reward follows then we must look at the motivations behind the dismissal. The only motivation offered by the text is that the boy is so good. But other, hidden motivations may be behind this statement. We have to look into psychology. Being ‘so good’ can be seen as a compliment and maybe this compliment is given to cheer up the boy. He may have been dismissed for other reasons (such as being a bad team player (in my mail to Jari I used the football player Christiano Ronaldo as an example)) but the dean and the teachers choose to be kind to the boy and stress the fact that he is good. Maybe other players on the team felt ill at ease with so much talent around and the team, rather than the excellent player, had to be saved.

There is a world of possibilities behind the inner workings of the meeting of the dean and the teachers’ council. There are many things that the text does not reveal; we only see the curtailed conclusion. Within the council there may have been a teacher who felt that the boy’s grades suffered from playing ice hockey so much and pleaded for his dismissal. To Jari I even offered the suggestion that bets were placed on the ice jockey games (unlikely, but interesting for the sake of argument) and that the dean or the teachers had been bribed to make the team fail. The boy may or may not have parents who play a part in the decision. No parents are mentioned in the text and this is odd in some way because parents are usually involded in school issues.


If we want to apply analysis we must learn how to reason. We must learn how to recognize and analyze conclusions, reasonings, facts and assumptions. How do I judge the conclusion in the challenge? I know it is short and illogical in my world view.  There are numerous gaps in this conclusion and I will have to apply reasoning to locate them. How do I judge the entire challenge within the light of my limited modern Western world mindset. In the book The Order of Things the French philospher Michel Foucault tells us that reading the classification of animals stated in the Celestial Emporium of Benevolent Knowledge (by the Argentine writer Jorge Luis Borges) broke “up all the ordered surfaces and all the planes with which we are accustomed to tame the wild profusion of existing things”. He further argues that a categorization changes when you change the background (context) of that classification. The concept, of course, is simple: if you place a black square in a black background you will not even notice it is there. If you place it in a white background you will see a black square. Has the square changed?

Some other friends that accompany us during analysis are quantification, time and place (setting). From, for example, movies, we learn that time is not absolute, that sequences of actions do not have to be consistent in time, that time may be sped up, slowed down, reversed etc… Similarly any quantification, or lack of quantification may hide or reveal gaps. In the challenge the boy is 15 years old, he studies in a (any?) high school, there is a “teacher’s council”, how many teachers are in that council, do any of those teachers actually teach in the classes in which the boy participates?

Lately I’ve been thinking a lot about ‘what is missing‘ and what reasoning we can use to find what is missing. There is a nice book by the Austrian philosopher Paul Feyerabend with the rallying title Against Method. I cannot think of a reason why a software tester should not read this book. Feyerabend is all about context and he does nice things with Galileo Galilei. He uses Galilei’s reasoning in favour of the movement of the earth (around the sun). What Galilei did, according to Feyerabend, was to show something that was missing by reasoning. In  Galilei’s case the movement of the earth was rejected because a rock, falling from a tower, falls straight down. If the rock falls straight down it proves that the earth does not move. Galilei argued that there are movements that we cannot percieve, and by proving this demonstrated that we at least must use other means of perception to draw conclusions about the movement of the earth.

Within the history of science, philosophy,sociology, biology etc… there is a whealth of reasoning. I think that it can be demonstrated that the tester who can analyze from a broad base of reasoning adds immense value to any project.  So if we must learn anything to add value to our craft, it is this. I hope this is a falsifiable theory.