Who should write the test code?


I will be discussing this and other FitNesse related topics in my presentation Moving from Ad Hoc Testing to Continuous Test Data with FitNesse at the Agile Testing Days 2014. If you are interested, please visit my session in Potsdam.

Not so long ago I found myself in a debate in which my motives for doing the work that I do currently and the way I that work were questioned by a team member, a senior developer. As far as I was able to ascertain the questions were earnest and coming from someone who really cares deeply about the software product we are making. Therefore, I thought that I should mention my approach and the questions that arose and how I dealt with these questions.

Who should write the test code?

I am currently using FitNesse as a test automation tool. Writing tests in FitNesse involves writing so-called fixtures in Java. The fixtures arrange the interactions with the system under test, such as the execution of database queries, calls to stored procedures, REST interface calls, kicking off unix scripts, FTP calls or the reading of log files. As the test set grows so does the repository of Java code. The code is maintained by two testers who have some experience with Java, but no background as a Java developer. We have been able to get most of the code working correctly and the code base is still small enough to allow for the many insights for refactoring. So there is room to learn from our coding mistakes.

Now the source code of the system under test is in PL/SQL and all of the developers in the team are proficient in this language. None of them have any experience in programming in Java. So in our team there are two testers maintaining the Java test code base and a number of PL/SQL developers maintaining the code base of the system under test. The senior developer now put forward the following line of reasoning (this is not a literal quote).

The test code should be maintained in the same fashion as the production code. When a developer is doing rework on the production code he should also be handling the related test code. It is the only way that we can ensure that the test code remains consistent. Therefore the test code should be written by a developer and in PL/SQL.

This may sound like a very decent proposal. After all, perhaps testers should not be focusing on writing code, which is usually not their expertise. When developers write the test code, the testers can focus their energy on testing the software. This way the effort of  creating test code and testing is shared in the most efficient way among the skills sets present in the team.

Also, treating the test code as production code is a very laudable sentiment. It means that test code and production code are equally important, which in my opinion they are. Delivering a piece of production code without the proper means to test that code does not make a lot of sense if you want the production code to be properly tested.

While all of these statements reflect sentiments that may lead to better cooperation and easier maintenance of the system and the test automation effort, I have a number of objections that, in my opinion carry some weight against this approach. These objections are also the reason that I am putting off the transition from Java to PL/SQL for test code.

1. Deployment of test code

We, as testers, are able to deliver changes in the Java code to the test environment almost instantaneously. I created a build script in Ant and this build script runs in Jenkins every day. So at the very least every day all of the code that is used for test automation is compiled and deployed to the environment where the tests are run. If the code contains errors we know this immediately. Otherwise the code is compiled and deployed without us ever having to think about it. However; deployment of PL/SQL packages to the test and acceptance environment cannot be done in an instant and certainly isn’t automated. Test packages would follow the same deployment procedures as production code and for some reason this is a cumbersome process. If I want to add some functionality in the Java code, I write it and I deploy it instantaneously. If I wanted to do that same thing in PL/SQL, I’d have to wait for the functionality to arrive, at best within a couple of hours.

2. Testing the test code

Over the last couple of months I have grown very fond of JUnit. I still do not write unit test on a structural basis and the packages that contain the unit tests are still a mess that I should sort out. But I am using JUnit to at least run a couple of tests against my code and gain some confidence in the solutions that I build. So the question is; would the developers do the same thing and the code they write to a certain degree? I asked this question a couple of times and got mixed responses.

3. The framework is linked to certain languages

If PL/SQL packages were really used as test code, we would still have to call them using a Java layer. This layer would probably not be very complicated, but FitNesse would still require us to use Java. So there would then be two bases of test code; the PL/SQL packaged and the Java code calling the packages.

4. Prioritization

If I am not writing the test code, someone else ought to be doing it (for me?). This other person may or may not like having to write test code (instead of working on production code). He may attach different priorities to test code. Also, if we would try to write user stories for building test code, these user stories may be deprioritized against user stories for pressing bug fixes or rewarding new functionality. Creating test code as production code requires discipline, a relentless commitment to the definition of done and a product owner who deeply feels our need to use automated tests. The current situation is too far away from that ideal for me to start this experiment with the code that I depend on to generate test data.

5. It is not at all bad for testers to know the test code

I think it can be argued that testers should know intimately the code that they use to automate their tests. Automated tests generate results. If a tester does not know where the results are coming from, or how they are fabricated, how can he or she trust them?

6. Software developers have a different take on testing than testers

Test code has to have the capability to execute a number of scenario’s, it requires flexibility and it needs to be designed for testing, not for production. I believe that testers have a better feeling for how they want to use the test code. I believe, for example, that they more easily think in terms of scenarios and more easily come up with variations than do developers. And so testers may be better at indicating what the test code should be able to handle.

Exploring FitNesse – Using Database Queries


In this series of blog posts I want to explore my choice to use the Java Persistence API in a test automation effort against a legacy (Oracle) database. For test automation I currently use FitNesse. Fitnesse offers many ways to access information from the database and for those who do not like to get their hands dirty writing Java (or other) fixtures, the Java Persistence API is definitely not the way to go.

Like I said, FitNesse offers many ways to access the database from the FitNesse page or from somewhere just below the FitNesse page, with an absolute minimal amount of (Java) programming. Below I list a couple of options. I never worked actively with any of these options, but I am familiar with them from reading through the documentation.

  • DbFit by Goijko Adzic. This seems to be a viable alternative to placing SQL queries in the Java layer. Adzic’ fixture set seems to be pretty complete and I recently learned that it is actually used in a data warehousing project in the Netherlands.
  • JdbcFixtures. This set of fixtures is limited and probably it can only be used for basic stuff.
  • A generic FitNesse fixture that is ready to use. The fixture mentioned was written by Anubhava Srivastava. It is perfectly fine to use code that someone else offers for use. But you’d have to know programming to check the code, adjust it to your needs or develop it further.
  • FitNesse QueryTable. This requires you to write the actual query in the Java fixture, so (a bit of) programming is involved. The table itself offers a nice way to return more than one row of data and check for certain outcomes.

Three of the options listed above allow for the tester to write the actual SQL statements in the FitNesse page. The tester does not have to know a lot about drivers and connections to write tests that fetch data from the database or even manipulate data that is in the database. The statements are executed ‘under water’ and the result is displayed in the FitNesse page. It is a nice way to quickly start using SQL queries and to keep all the logic in the same place, namely the FitNesse page.

I dismissed the use of SQL statements in FitNesse pages rather quickly after I started on the project. We had some pages in which SQL statements were used. In my opinion SQL statements were technical (implementation) details and I wanted to place all technical details in the layer that, according to me, they belong in – which is the programming (in my case Java) layer.

Besides that I thought that scattering many SQL statements across many FitNesse pages would seriously impact the maintainability of statements should the developers ever decide to change some of the tables. Actually there is a solution for the maintainability problem: build modular tests (building blocks) in FitNesse. If a query is used by many pages, build a separate FitNesse page for this query only, parametrize the page using variables and include (FitNesse: !include) it in the pages that use the query. I use the modular approach in FitNesse a lot and it works brilliantly. I will write more about this later.

A second objection to writing SQL statements in FitNesse pages is that it would probably take a while to find out which table is queried in which page, in what way. FitNesse has reasonable search capacities, but having to search for queries all the time is not ideal. The modular approach outlined above may fix this issue partly.

All in all I still had five major objections to placing SQL statements in FitNesse pages.

  1. Maintainability. This I explained above.
  2. Testability. I like to test my queries 🙂 Typos are quickly made and sometimes not that easy to find. Complex joins also invite mistakes.
  3. Handling of exceptions. Some queries may return nothing (no rows or null) or errors. FitNesse may not be the proper tool to handle these exceptions.
  4. Managing connections. Again, there are better ways to manage connections than in FitNesse pages.
  5. Handling of types. I am not sure if this is actually a problem. Fields in the database can be of many types (VARCHAR, FLOAT, DATE, TIMESTAMP, LONG, CHAR,  BLOB etc…). I think FitNesse ‘represents’ everything as text. I think this works out ok for non-text types, but I haven’t tested it. In Java, when fetching different types of data from the database, the type is something you carefully have to pay attention to.

So I decided to embed SQL statements in the Java layer (the Java fixtures) that is beneath FitNesse. However, here the maintainability problem remains. Sure, the testing of queries is made much easier by using jUnit. Connections can definitely be managed better in the Java layer. And Java has tools to handle exceptions programmatically. But you still run the risk of scattering SQL queries in all shapes and sizes all across the code.

This last observation eventually led me on the path to the Java Persistence API, about which I will write in a follow-up blog post.