A Systems Perspective on Testing

Standard

Lately I’ve been looking at Systems theory (Bertalanffy, 1968) and wondering how much of it has been applied to testing and how much more could still be learned from it.

Three blind spots

I started with the assumption that little, if any, of the concepts in Systems theory have been applied in testing. This assumption proved to be false.  There were a number of blind spots in my assumption. I will list the most prominent of them.

1) Little explicit mention of Systems theory in testing material

The first blind spot arose from the observation that Systems theory is hardly ever explicitly mentioned or referenced in day-to-day testing. By day-to-day testing I mean the insights provided by the text books as used for certification (ISTQB, TMap), the testing standards, blogs on testing, articles on testing as published in the popular testing magazines and the mindset and knowledge of software testers. Although software testing deals, almost exclusively, with systems, there seems to be no explicit connection with Systems theory, or concepts within this field. To account for this, we should consider the possibility that software testing in itself is a subset of computation which is a subset of Systems theory. As such software testing may look inward for more detail, instead of outward for a position in the general picture.

2) Little knowledge of Systems theory among testers

The second blind spot is the knowledge, or common perception, among testers of the field of Systems theory. Within the context-driven school of testing, there are testers that look beyond the traditional views offered by certification institutes or testing standards. So there is a willingness to learn about, among other things, systems theory. Yet as a starting point for Systems theory (or Systems thinking) often the book Thinking in Systems (Meadows, 2008) is mentioned. Meadows wrote about System dynamics (Forrester, 1961), which certainly is a part of Systems theory, but it would be a gross overstatement that Meadows covers the landscape of Systems theory in total. A personal investigation of Systems theory revealed  – to me it was a revelation – that there are at least 18 fields of scientific investigation that are related to Systems theory. System dynamics is only a fragment.

3) The academic-vocational chasm in software testing

The third thing that obscured my perception of Systems theory and testing, is that software testers in the field are largely unaware of the theoretical (scientific, academic) foundations of software testing. I consider this a problem that affects the industry as a whole. There is a curious separation between the work that (academic) researchers in software testing do, and what actually filters down to the work floor. This may be a common phenomenon that is not at all particular to software testing, but it frequently frustrates me that what is taught in testing text books and testing courses is a filtered, abstracted and sometimes disfigured version of the original theories on software testing. Particularly of the 1970’s, which was a period in which software testing theory was formed, the tester in the field generally knows very little, perhaps only a few quotes, cut out of their context.

System theory in software testing

So if we look at how software testing theory evolved in the 1970’s and 1980’s, it becomes clear that Systems theory – certain (mathematical) system perspectives applied to, or originating from(!), computing and software – has been the main driver for that effort.

Automata theory

One of the branches of Systems theory is the Automata theory. Automata theory by and large started with Alan Turing, who conceived the Turing machine in 1936. From this theory arose theories on computational complexity, which is itself is a branch of Complexity theory, which is a branch of Systems theory. In computing Automata theory gave rise to finite state machines, for which we have a test technique called state transition testing (Chow, 1978).

Graph theory and partition analysis

Other ways of looking at systems, closely related to system design, developed in software testing in the 1970’s. Theories formed about the flow of information through software code and the selection of test data for testing the program. Regarding the flow of information Graph theory (branch of Network theory, branch of Systems theory)  was frequently used. The cause-effect graph (Elmendorf, 1973) is an example of that knowledge applied to in software testing. Also using Graph theory several path-based techniques were introduced, such as basic path testing (McCabe, 1976) and decision-to-decision path testing (Miller, 1977). One other (domain testing) technique, partition analysis (or equivalence partitioning), was particularly used for the selection of data to be used to test the paths (Goodenough, 1975). Partition analysis can be thought of as an application of Set theory, which is a branch of Systems theory.

Hierarchy and modularity

Another way of looking at systems is by seeing them as a hierarchical structure. This is called Hierarchy theory. On software development, the concept of hierarchical structures has had a huge influence. In 1972 David Lorge Parnas introduced decomposition and modularity into programming (Parnas, 1972). In testing the functional decomposition is a way to reduce the complexity of a system.

 Applying Systems theory

The problem with applying insights from Systems theory to software testing is a manifold problem. The first question that needs to be tackled is whether the making of software has changed in such a way that a new technique is warranted. In short: do we need other test techniques to test the current software systems? I ask this question because a number of our ‘black box’ testing techniques (particularly path testing and partition testing) definitely come from a ‘white box’ background. They are closely related to the way software (a program) is made. It therefore should be no surprise that software testing started out as ‘program testing’ (Hetzel, 1973). Other techniques, such as CRUD analysis (Martin, 1981) are equally based on code. Has the coding (programming) of our systems changed in such a way that the old techniques are no longer adequate to cover the system sufficiently? Has a change in programming forced upon us the usage of other aspects of Systems theory in order to create new techniques? Looking at our testing text books, I still notice a certain prominence of old techniques. Perhaps the answer to this question should be ‘no’.

There are two other questions that may be more interesting if we want to use the Systems theory perspective in software testing. The questions are: has the behavior of software changed and has the usage of software changed? Perhaps there are fundamentally no radical changes in code, but there may be radical changes in behavior or usage of software systems that could eventually lead to other ways of failure than those investigated using the classic test techniques. Perhaps there are patterns in systems that cannot be investigated using partition or path analysis.

I think it is safe to state that software usage has changed radically and that – accordingly – in the functioning of software behavioral patterns have emerged that require us to rethink the tools we use to test the behavior or functioning of systems. It is likely that Systems theory can help us find these tools. In subsequent posts I will try to investigate these new tools.

References

Bertalanffy, Ludwig von  – General System Theory (1968)

Chow, Tsun S. – Testing Software Design Modeled by Finite-State Machines (1978)

Elmendorf, William – Cause-Effect Graphs in Functional Testing (1973)

Forrester, Jay – Industrial Dynamics (1961)

Goodenough, John B. – Toward a Theory of Test Data Selection (1975)

Hetzel, William – Program Test Methods (1973)

Martin, James – Information Engineering (1981)

McCabe, Thomas J. – A Complexity Measure (1976)

Meadows, Donella H. – Thinking in Systems (2008)

Miller, Edward F. – Program Testing Techniques (1977)

Parnas, David Lorge – On the Criteria To Be Used in Decomposing Systems into Modules (1972)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s