Who should write the test code?

I will be discussing this and other FitNesse related topics in my presentation Moving from Ad Hoc Testing to Continuous Test Data with FitNesse at the Agile Testing Days 2014. If you are interested, please visit my session in Potsdam.

Not so long ago I found myself in a debate in which my motives for doing the work that I do currently and the way I that work were questioned by a team member, a senior developer. As far as I was able to ascertain the questions were earnest and coming from someone who really cares deeply about the software product we are making. Therefore, I thought that I should mention my approach and the questions that arose and how I dealt with these questions.

I am currently using FitNesse as a test automation tool. Writing tests in FitNesse involves writing so-called fixtures in Java. The fixtures arrange the interactions with the system under test, such as the execution of database queries, calls to stored procedures, REST interface calls, kicking off unix scripts, FTP calls or the reading of log files. As the test set grows so does the repository of Java code. The code is maintained by two testers who have some experience with Java, but no background as a Java developer. We have been able to get most of the code working correctly and the code base is still small enough to allow for the many insights for refactoring. So there is room to learn from our coding mistakes.

Now the source code of the system under test is in PL/SQL and all of the developers in the team are proficient in this language. None of them have any experience in programming in Java. So in our team there are two testers maintaining the Java test code base and a number of PL/SQL developers maintaining the code base of the system under test. The senior developer now put forward the following line of reasoning (this is not a literal quote).

The test code should be maintained in the same fashion as the production code. When a developer is doing rework on the production code he should also be handling the related test code. It is the only way that we can ensure that the test code remains consistent. Therefore the test code should be written by a developer and in PL/SQL.

This may sound like a very decent proposal. After all, perhaps testers should not be focusing on writing code, which is usually not their expertise. When developers write the test code, the testers can focus their energy on testing the software. This way the effort of  creating test code and testing is shared in the most efficient way among the skills sets present in the team.

Also, treating the test code as production code is a very laudable sentiment. It means that test code and production code are equally important, which in my opinion they are. Delivering a piece of production code without the proper means to test that code does not make a lot of sense if you want the production code to be properly tested.

While all of these statements reflect sentiments that may lead to better cooperation and easier maintenance of the system and the test automation effort, I have a number of objections that, in my opinion carry some weight against this approach. These objections are also the reason that I am putting off the transition from Java to PL/SQL for test code.

1. Deployment of test code

We, as testers, are able to deliver changes in the Java code to the test environment almost instantaneously. I created a build script in Ant and this build script runs in Jenkins every day. So at the very least every day all of the code that is used for test automation is compiled and deployed to the environment where the tests are run. If the code contains errors we know this immediately. Otherwise the code is compiled and deployed without us ever having to think about it. However; deployment of PL/SQL packages to the test and acceptance environment cannot be done in an instant and certainly isn’t automated. Test packages would follow the same deployment procedures as production code and for some reason this is a cumbersome process. If I want to add some functionality in the Java code, I write it and I deploy it instantaneously. If I wanted to do that same thing in PL/SQL, I’d have to wait for the functionality to arrive, at best within a couple of hours.

2. Testing the test code

Over the last couple of months I have grown very fond of JUnit. I still do not write unit test on a structural basis and the packages that contain the unit tests are still a mess that I should sort out. But I am using JUnit to at least run a couple of tests against my code and gain some confidence in the solutions that I build. So the question is; would the developers do the same thing and the code they write to a certain degree? I asked this question a couple of times and got mixed responses.

3. The framework is linked to certain languages

If PL/SQL packages were really used as test code, we would still have to call them using a Java layer. This layer would probably not be very complicated, but FitNesse would still require us to use Java. So there would then be two bases of test code; the PL/SQL packaged and the Java code calling the packages.

4. Prioritization

If I am not writing the test code, someone else ought to be doing it (for me?). This other person may or may not like having to write test code (instead of working on production code). He may attach different priorities to test code. Also, if we would try to write user stories for building test code, these user stories may be deprioritized against user stories for pressing bug fixes or rewarding new functionality. Creating test code as production code requires discipline, a relentless commitment to the definition of done and a product owner who deeply feels our need to use automated tests. The current situation is too far away from that ideal for me to start this experiment with the code that I depend on to generate test data.

5. It is not at all bad for testers to know the test code

I think it can be argued that testers should know intimately the code that they use to automate their tests. Automated tests generate results. If a tester does not know where the results are coming from, or how they are fabricated, how can he or she trust them?

6. Software developers have a different take on testing than testers

Test code has to have the capability to execute a number of scenario’s, it requires flexibility and it needs to be designed for testing, not for production. I believe that testers have a better feeling for how they want to use the test code. I believe, for example, that they more easily think in terms of scenarios and more easily come up with variations than do developers. And so testers may be better at indicating what the test code should be able to handle.