Solving a Ten Thousand Piece Puzzle


On the third of March a meeting was organized by Improve Quality Services (my employer) and the Federation of Agile Testers in Utrecht, the Netherlands. The evening featured James Bach as speaker and his talk focused on the paper A Context-Driven Approach to Automation in Testing, which was written by him and Michael Bolton. My favorite part of the evening was the part during which James tested some functionality of an application and explains his way of working. He provided such a demonstration a year ago when introducing the test autopsy.

The exercise

inkscape_logoThis time around the subject under test was the distribution function of the open source drawing tool Inkscape and the focus was on the usage of tools to test this functionality. It must be said that Inkscape lends itself to the usage of tools because it stores all the images that are generated using this tool in the Scalable Vector Graphics (SVG) format, which is an open standard developed by the World Wide Web Consortium (W3C). This greatly increases the intrinsic testability (links opens PDF) of the product, as we will see.

The SVG format is described in XML and as such, the image is a text file that can be analyzed using different tools. It is also possible to create text files in the SVG format that can then be opened and rendered in Inkscape. As such, the possibilities for creating images by generating the XML script using code are virtually limitless. Before the start of the presentation James had created a drawing containing 10,000 squares. He created this drawing using some script (I am not sure he mentioned in which language this script was written). My initial reaction to James showing the drawing that he generated was one of astonishment. I was impressed by his idea of testing this functionality with 10,000 squares, by the drawing itself and the fact that it was generated using a script.

Impressed by complexity

Looking back, my amazement may have been caused my lack of experience with Inkscape and the SVG format. But it also reminded me that it is easy to be impressed by something new and especially if this new thing seems to be complex. I believe that, in testing, if you really want to impress people — for all the wrong reasons — all you need to do is to present to them a certain subject as being complex. People will revere you because you are the only one who seems to understand the subject. The exercise, as James walked us through it, seemed complex to me and this is what triggered me to investigate it.

So why use ten thousand squares?

I am sure it was not James’ intention to impress us, so then the question is: why would he use ten thousand squares? Actually this question occurred to me halfway through doing the exercise myself, when tinkering with the distribution function. Distributing, for example, 3 squares is easy; it does not require a file generated with a script. Furthermore, it is easy to draw conclusions from the distribution of 3 squares. Equal distribution can be ascertained visually (by looking at the picture), with a reasonable degree of certainty. So if equal distribution functions correctly with 3 squares, why would there be a problem with 10,000 squares? I am assuming that the distribution algorithm does not function differently based on the amount of input. I mean, why would it? So, taking this assumption into account, using 10,000 squares during testing does only the following things:

  1. It complicates testing, because it is no longer possible to ascertain equal distribution visually.
  2. Because of this, it forces the tester to use tools to generate the picture and to analyze the results.
  3. It complicates testing, because the loading of the large SVG file and the distribution function take a significant amount of time.
  4. It tests the performance of the distribution function in Inkscape.

Now the testing of the performance is not something I want to do as a part of this test. But it seems that working with 10,000 squares adds something meaningful to the exercise. A distributed image generated from 10,000 squares does not allow for a quick visual check and therefore simulates a degree of complexity that requires ingenuity and the use of tools if we want to check the functioning of distribution. Working with large data sets and having to distill meaning from a large set is, I believe, a problem that testers often face. So, as an exercise, it is interesting to see how this can be handled.

A deep dive into the matter

Some of the tools I use

  • Inkscape (for viewing and manipulating images)
  • Python (for writing scripts)
  • Kate (for editing scripts and viewing text files)
  • KSnapshot (for creating screen shots)
  • Google (for looking up examples & info)
  • R (for statistical analysis)

In an attempt to reproduce James’ exercise, I create a script to generate this drawing myself. In order to do so, I need to find out a little bit about the SVG standard. Then I create an Inkscape drawing containing one square, in order to find out the XML format of the square. Now I have an SVG file that I can manipulate so I have enough to start scripting. I install Python on my old Lubuntu netbook which is easy to do. I never did much programming in Python before. I could have written the script in PHP or Java, which are the two programming languages about which I know a fair amount, but it seems to me that Python is fairly light-weight and suitable for the job. It can be run from the command line without compilation, which contributes to its ease of use.

So I write a Python script that creates an SVG file with 10,000 squares in it. Part of the script is displayed below. I look up most of the Python code by Googling it and copy-pasting from examples, so the code is not written well, but it works. I can run the script from the command line and it generates the file in the blink of an eye. The file size of the file is just about 2.4 Mb, which is fairly large for a text file and when I open it using Inkscape, the program becomes unresponsive for a couple of seconds. Apparently the program has some difficulty generating the drawing, which is understandable, given that the file is large and the netbook on which I run the application is limited in both processing power and internal memory (2 Gb). Yet, the file opens without errors and shows a nice grid of 10,000 squares.

Python script for creating the squares

with open('many_squares.svg', 'w') as f:
    offset = 12
    number_of_squares = 10000

    while number_of_squares > 0:
        square = '''<rect
        y="%d" />''' % (x,y)
        if x + offset > 800 :
            y = y + offset
            x = x + offset
        number_of_squares = number_of_squares -1

Which results in the following picture.

ten thousand squares grid

The regular grid of 10,000 squares created with the Python script

ten thousand squares grid close up

A close up of the grid created with the Python script


inkscape - align and distribute

The Inkscape distribution functions

I now have a grid of 10,000 squares with which I am trying to reproduce James’ exercise. The thing that I run into is that Inkscape has a number of distribution options. I am not sure which distribution James applied, so I try a couple. None of them however show as a final result the image that James showed during his presentation – as far as I can remember it was an oval shape with a higher density of objects near the edges. Initially it seems strange that I am unable to reproduce this, but through tinkering with the distribution function, I conclude that the fact that I am unable to reproduce James’ distributed image probably depends on the input. The grid that I create with the script contains identical squares of 10 by 10 pixels, evenly spaced (12 pixels apart) along the x and the y axes. It may differ in many aspects (for example, size, spacing and placement of the objects) from the input that James created.

Developing an expected result

I apply the Inkscape distribution functionality (distribute centers horizontally and vertically) to my drawing containing the 10,000 squares and the result is as shown below. The resulting picture looks somewhat chaotic to me. I cannot identify a pattern and even if I could identify a pattern, I would not be sure if this pattern is what I should be seeing. There seem to be some lines running through the picture, which seems odd. But in order to check the distribution properly, I need to develop an expected result, using oracles, by which I can check if the distribution is correct.

ten thousand squares distributed

The entire distributed drawing

ten thousand squares distributed close up

A close up of the distributed drawing (it kind of looks like art)


I do several things to arrive at a description of what distribution means. First I consult the Inkscape manual with regards to the distributions functions that I used. The description is as follows.

Distribute centers equidistanly horizontally or vertically

Apart from the spelling mistake in the manual, the word that I want to investigate is ‘equidistant’. It means — according to Merriam-Webster —

of equal distance : located at the same distance

Distance is complex concept. The Wikipedia page on distance is a nice starting point for the subject. I simplify the concept of distance to suit my exercise by assuming a couple of things. My definition is as follows: Distance is the space between two points, expressed as the physical length of the shortest possible path through space between these points that could be taken if there were no obstacles. In short, the distance is the length of the path in a straight line between two points.

There are other things I need to consider. The space in which the drawing is made is two dimensional. This might seem obvious, but it is important to realize that every single point in the picture can be identified with a two dimensional Cartesian coordinate system. In short, every point has a x and a y coordinate (which we already saw when generating the SVG file) and this realization greatly helps me when I try to analyze the picture.  Another question I need to answer is which two points I use. This is tricky, because in my exercise, I used the center of the square as reference point for distribution. Since I am dealing with squares, width and height are equal and since all the squares in my drawing have the same width and height, I simplified my problem in such a way that I can use the x an y coordinates of the top left corner (which can be found in the SVG file) of each square for further analysis. There is no need for me to calculate the center of each object and do my analysis on those coordinates.

And lastly I need to clarify what distribution means. It turns out that there are at least two ways to distribute things. I came across an excellent example in a Stack Exchange question. In this question the distinction is made between spreading out evenly and spacing evenly. To spread out evenly means that the centers of all objects are distributed evenly across the space. To space evenly means that the distance between the objects is the same for all objects. The picture below clarifies this.

Types of distribution

Types of distribution (source: Stack Exchange)

In my special case — I am working exclusively with squares that are all the same size — to spread out evenly means to space evenly. So the distinction, while relevant when talking about distribution, matters less to me. Aside from the investigation described above, I spoke with several co workers about this exercise and they gave me some useful feedback on how I should regard distribution.

To make a long story short, my expected result is as follows.

Given that all the objects in the drawing are squares of equal size, if the centers of all the squares are distributed equally along the x axis then I can analyze the x coordinates of the top left of all squares. If the x coordinates are sorted in ascending order, I should find that difference between one x coordinate and the x coordinate immediately following it, is the same for all x coordinates. The same should go for the y coordinates (vertical distribution).

This is what I’m looking for in the drawing with the distributed squares.

Some experiments in R

In order to do some analysis, I need the x and y coordinates of the top left corner of all the squares in the drawing. It turns out to be fairly easy to distill these values from the SVG file using Python. Again, I create a Python script by learning from examples found on the internet. The script, as displayed below, subtracts from the SVG file the x and y coordinates of the top left corner of each square and then writes these coordinates to an comma separated file (csv). The csv file is very suitable as input for R.

Python script for generating the csv file containing the coordinates

svg = open("many_squares_distr.svg", "r")

coordinates = []
for line in svg:
    print line
    if line.find(' x=') <> -1 or line.find(' y=') <> -1: # line containing x or y coordinate found
        positions = []
        for pos, char in enumerate(line):
            if char == '"':
        print line[positions[0]+1:positions[1]]
        if line.find('x=') <> -1 :
            x = line[positions[0]+1:positions[1]]
        if line.find('y=') <> -1 :
            y = line[positions[0]+1:positions[1]]
            new_coordinate = [x,y]

with open('coordinates.csv', 'w') as f:
    point = 1
    line = line = 'X,Y\n'
    for row in coordinates:
        line = '%s,%s\n' % (row[0],row[1])
        point = point + 1

Now we come to the part that is, for me, the toughest part of the exercise on which I consequently spent the most time. The language that I use for the analysis of the data is R. This is the language that James also used in his exercise. The language is entirely new to me. Furthermore, R is a language for statistical computation and I am not a hero at statistics. What I know about statistics dates back to a university course that I took some twenty years ago. So you’ll have to bear with the simplicity of my attempts.

It is not difficult to load the csv file into R. It can be done using this command.

coor <- read.csv(file="coordinates.csv",head=TRUE,sep=",")

A graph of this dataset can be created (plotted).


Resulting in the picture below.

plotted x and y
After that, I am creating a new dataset that only contains the x values, using the command below.

xco = coor[,1]

And then I sort the x values ascending.

xcor <- sort(xco)

Then I use the following command


to create a graph of the result as displayed below.

plotted sorted x coordinates

The final result is a relatively perfectly straight line because (as I expected) for each data point the (x) value is increased by the same amount, resulting in a lineair function. As a result this satisfies my needs, so this is where I stop testing. I could have created, using a Python script, a dataset containing all the differences between the consecutive x coordinates and I could have checked the distribution of these differences with R. I leave this for another time.


One of the questions you might ask is if I really tested the distribution functionality. My answer would be a downright ‘No’. I used the distribution functionality in an exercise, but the goal of the exercise was not to test the functionality. The goal was to see what tools can do in a complex situation. If I had really investigated the distribution functionality, I would have created a coverage outline and I would certainly have tried different kinds of input. Also I would have had to take a more in depth look at the concept of distribution.

One of the results of this exercise is that I know a little bit more about scripting, about the language R and about vector images. Also, I learned that the skills related to software testing are manifold and that it is not easy to describe them. I particularly liked describing how I arrived at my definition of the expected result, which meant investigating different sources and drawing conclusions from that investigation. I feel that the software tester should be able to do such an investigation and to build the evidence of testing on it. I also learned again that complexity is a many headed monster that often roams freely in software development. Testers need to master the tools that can tame that beast.

Some exploration took place in the form of ‘tinkering’ with the distribution function of Inkscape. This helped me build a mental model of distribution. Furthermore I toyed with R on a ‘trail and error’ basis, in order to find out how it works.


Agile Testing Days 2015 – At the Lab


This year I attended the Agile Testing Days for the second time and I can say for sure that I will be attending in 2016. It is worth to return to this conference, because it has as delightful mix of topics across the Agile spectrum. The program offers technical talks and workshops that focus on coding, debugging, frameworks and test automation. It also features talks and workshops on Agile leadership and collaboration, with focus on human aspects, motivation and learning. And if offers sessions on many aspects of testing, such as exploratory testing, note-taking, modeling and the relation of testing to Agile methodologies. There is more than enough to go round for the modern tester.



From this year’s conference I took away some particular things. I spend quite some time in the Anything Build Party & TestLab, expertly hosted by James Lyndsay and Bart Knaack. I played around with CodeBug, which is a programmable and wearable device with 25 leds (see picture). The device can be programmed using a Blockly-based programming interface, which makes it easy to build controls for the LEDs. I spend a while thinking about what I would like to create and then got stuck halfway through coding. At that point I paired up with another participant trying to improve the program that I wrote. We seemed to run into the limits of the programming interface, but it was fun doing this together and we were both energized by the experience.

I also finally took a serious look at some black box puzzles, created by James Lyndsay. At the Belgium Testing Days last year I struggled with a black box puzzle, turned into a physical puzzle by Altom. Now I wanted to look at the other puzzles. With some hints from James I found a satisfying description of the functional behavior of puzzle 8. After that success I picked up puzzle 7 which I was unable to fully figure out, also because the conference was drawing to an end.

I learned the following things from the puzzles.

  • Taking notes helps. Using the notes as a guideline, it is easier to explain your testing. At least, that was how I was able to explain my testing when people who stopped by asked what I had been testing so far. Taking notes in a notebook with a pen or pencil is quicker, much more flexible and much more intuitive than taking notes in the form of a mind map. I tried both the mind map and the pencil. Creating a mind map (even tough it appears to be very light weight tool), seemed to interrupt the flow of thoughts and the flow of testing exactly when I needed that flow. I chucked the mind map and grabbed the pencil. I always thought of a mind map as a ‘informal’ tool that enables creativity. I do so now to a lesser degree.Black box puzzle 8
  • Having a mental model helps you devise theories and experiments. The mental model is the idea you have about how the subject under test might function. Whether the idea is right or wrong doesn’t matter that much. I got pretty far in explaining the behavior of puzzle 8 using a wrong (but very useful!) mental model. The beauty is that once you have an idea about how an application might function, you can test that theory. Often this provides you with more information about its behavior and about the validity of your model. Having gained that information I was able to move to a more advanced model. I think I moved through several mental models during puzzle 7 and 8.
  • Frequent interaction with the subject under test is important. With regards to the puzzles I tried, I found it not very useful to sit back and philosophize a whole lot about the excercise. Reflection is needed in order to move ahead through the theories about the functioning of the device, but you need data to reflect on. Interacting with the subject under test makes you notice certain behavior and it will generates test data. This is the data that you need in order to build a theory. I observed people playing the dice game during the Agile Games Market on Wednesday evening and one participant in this game took long pauses (of up to a minute!) between ‘throws’. I was almost yelling ‘Come on, test, test, test!!’ to urge him to gather data faster. The brain needs something to work with.
  • And yet, taking a break helps. This has been mentioned time and again in testing literature and blog posts. It is true that when one detaches himself from the situation, usually the mind starts offering clues as to how the application functions. I took a cigarette break and it helped me. James told me a story about a man who tried one of his puzzles at a conference and got stuck. During the ensuing keynote he had an epiphany, went back to the lab and presented, in one go, the answer to the puzzle.
  • Use tools. Since puzzles 7 & 8 are about timing, it is nice to have a timer and use it. I took my smartphone and used the timer to at least get an idea of the time between the flashing of the red light. It helped me to be a bit more confident about the reproducibility of certain situations and it helped me to falsify (or confirm) my theories. Yet at my current assignment quite a number of testers seem to be very wary of using tools, whether it be a SQL query tool, a tool to transfers loan data from one database to another, to make notes, to call a SOAP interface, to view logs, to drive a GUI, to debug the environment etc… I think it is inexperience in using such tools that is keeping them.

Since I didn’t ‘solve’ puzzle 7, James gave me a final hint and he urged me to look at it at home. I will do this, but in the meantime I want to thank James for creating these puzzles, and James and Bart for running the testlab at so many conferences, making it possible for testers to reflect on their testing and to learn from it.

Who should write the test code?


I will be discussing this and other FitNesse related topics in my presentation Moving from Ad Hoc Testing to Continuous Test Data with FitNesse at the Agile Testing Days 2014. If you are interested, please visit my session in Potsdam.

Not so long ago I found myself in a debate in which my motives for doing the work that I do currently and the way I that work were questioned by a team member, a senior developer. As far as I was able to ascertain the questions were earnest and coming from someone who really cares deeply about the software product we are making. Therefore, I thought that I should mention my approach and the questions that arose and how I dealt with these questions.

Who should write the test code?

I am currently using FitNesse as a test automation tool. Writing tests in FitNesse involves writing so-called fixtures in Java. The fixtures arrange the interactions with the system under test, such as the execution of database queries, calls to stored procedures, REST interface calls, kicking off unix scripts, FTP calls or the reading of log files. As the test set grows so does the repository of Java code. The code is maintained by two testers who have some experience with Java, but no background as a Java developer. We have been able to get most of the code working correctly and the code base is still small enough to allow for the many insights for refactoring. So there is room to learn from our coding mistakes.

Now the source code of the system under test is in PL/SQL and all of the developers in the team are proficient in this language. None of them have any experience in programming in Java. So in our team there are two testers maintaining the Java test code base and a number of PL/SQL developers maintaining the code base of the system under test. The senior developer now put forward the following line of reasoning (this is not a literal quote).

The test code should be maintained in the same fashion as the production code. When a developer is doing rework on the production code he should also be handling the related test code. It is the only way that we can ensure that the test code remains consistent. Therefore the test code should be written by a developer and in PL/SQL.

This may sound like a very decent proposal. After all, perhaps testers should not be focusing on writing code, which is usually not their expertise. When developers write the test code, the testers can focus their energy on testing the software. This way the effort of  creating test code and testing is shared in the most efficient way among the skills sets present in the team.

Also, treating the test code as production code is a very laudable sentiment. It means that test code and production code are equally important, which in my opinion they are. Delivering a piece of production code without the proper means to test that code does not make a lot of sense if you want the production code to be properly tested.

While all of these statements reflect sentiments that may lead to better cooperation and easier maintenance of the system and the test automation effort, I have a number of objections that, in my opinion carry some weight against this approach. These objections are also the reason that I am putting off the transition from Java to PL/SQL for test code.

1. Deployment of test code

We, as testers, are able to deliver changes in the Java code to the test environment almost instantaneously. I created a build script in Ant and this build script runs in Jenkins every day. So at the very least every day all of the code that is used for test automation is compiled and deployed to the environment where the tests are run. If the code contains errors we know this immediately. Otherwise the code is compiled and deployed without us ever having to think about it. However; deployment of PL/SQL packages to the test and acceptance environment cannot be done in an instant and certainly isn’t automated. Test packages would follow the same deployment procedures as production code and for some reason this is a cumbersome process. If I want to add some functionality in the Java code, I write it and I deploy it instantaneously. If I wanted to do that same thing in PL/SQL, I’d have to wait for the functionality to arrive, at best within a couple of hours.

2. Testing the test code

Over the last couple of months I have grown very fond of JUnit. I still do not write unit test on a structural basis and the packages that contain the unit tests are still a mess that I should sort out. But I am using JUnit to at least run a couple of tests against my code and gain some confidence in the solutions that I build. So the question is; would the developers do the same thing and the code they write to a certain degree? I asked this question a couple of times and got mixed responses.

3. The framework is linked to certain languages

If PL/SQL packages were really used as test code, we would still have to call them using a Java layer. This layer would probably not be very complicated, but FitNesse would still require us to use Java. So there would then be two bases of test code; the PL/SQL packaged and the Java code calling the packages.

4. Prioritization

If I am not writing the test code, someone else ought to be doing it (for me?). This other person may or may not like having to write test code (instead of working on production code). He may attach different priorities to test code. Also, if we would try to write user stories for building test code, these user stories may be deprioritized against user stories for pressing bug fixes or rewarding new functionality. Creating test code as production code requires discipline, a relentless commitment to the definition of done and a product owner who deeply feels our need to use automated tests. The current situation is too far away from that ideal for me to start this experiment with the code that I depend on to generate test data.

5. It is not at all bad for testers to know the test code

I think it can be argued that testers should know intimately the code that they use to automate their tests. Automated tests generate results. If a tester does not know where the results are coming from, or how they are fabricated, how can he or she trust them?

6. Software developers have a different take on testing than testers

Test code has to have the capability to execute a number of scenario’s, it requires flexibility and it needs to be designed for testing, not for production. I believe that testers have a better feeling for how they want to use the test code. I believe, for example, that they more easily think in terms of scenarios and more easily come up with variations than do developers. And so testers may be better at indicating what the test code should be able to handle.

Exploring FitNesse – Using Database Queries


In this series of blog posts I want to explore my choice to use the Java Persistence API in a test automation effort against a legacy (Oracle) database. For test automation I currently use FitNesse. Fitnesse offers many ways to access information from the database and for those who do not like to get their hands dirty writing Java (or other) fixtures, the Java Persistence API is definitely not the way to go.

Like I said, FitNesse offers many ways to access the database from the FitNesse page or from somewhere just below the FitNesse page, with an absolute minimal amount of (Java) programming. Below I list a couple of options. I never worked actively with any of these options, but I am familiar with them from reading through the documentation.

  • DbFit by Goijko Adzic. This seems to be a viable alternative to placing SQL queries in the Java layer. Adzic’ fixture set seems to be pretty complete and I recently learned that it is actually used in a data warehousing project in the Netherlands.
  • JdbcFixtures. This set of fixtures is limited and probably it can only be used for basic stuff.
  • A generic FitNesse fixture that is ready to use. The fixture mentioned was written by Anubhava Srivastava. It is perfectly fine to use code that someone else offers for use. But you’d have to know programming to check the code, adjust it to your needs or develop it further.
  • FitNesse QueryTable. This requires you to write the actual query in the Java fixture, so (a bit of) programming is involved. The table itself offers a nice way to return more than one row of data and check for certain outcomes.

Three of the options listed above allow for the tester to write the actual SQL statements in the FitNesse page. The tester does not have to know a lot about drivers and connections to write tests that fetch data from the database or even manipulate data that is in the database. The statements are executed ‘under water’ and the result is displayed in the FitNesse page. It is a nice way to quickly start using SQL queries and to keep all the logic in the same place, namely the FitNesse page.

I dismissed the use of SQL statements in FitNesse pages rather quickly after I started on the project. We had some pages in which SQL statements were used. In my opinion SQL statements were technical (implementation) details and I wanted to place all technical details in the layer that, according to me, they belong in – which is the programming (in my case Java) layer.

Besides that I thought that scattering many SQL statements across many FitNesse pages would seriously impact the maintainability of statements should the developers ever decide to change some of the tables. Actually there is a solution for the maintainability problem: build modular tests (building blocks) in FitNesse. If a query is used by many pages, build a separate FitNesse page for this query only, parametrize the page using variables and include (FitNesse: !include) it in the pages that use the query. I use the modular approach in FitNesse a lot and it works brilliantly. I will write more about this later.

A second objection to writing SQL statements in FitNesse pages is that it would probably take a while to find out which table is queried in which page, in what way. FitNesse has reasonable search capacities, but having to search for queries all the time is not ideal. The modular approach outlined above may fix this issue partly.

All in all I still had five major objections to placing SQL statements in FitNesse pages.

  1. Maintainability. This I explained above.
  2. Testability. I like to test my queries 🙂 Typos are quickly made and sometimes not that easy to find. Complex joins also invite mistakes.
  3. Handling of exceptions. Some queries may return nothing (no rows or null) or errors. FitNesse may not be the proper tool to handle these exceptions.
  4. Managing connections. Again, there are better ways to manage connections than in FitNesse pages.
  5. Handling of types. I am not sure if this is actually a problem. Fields in the database can be of many types (VARCHAR, FLOAT, DATE, TIMESTAMP, LONG, CHAR,  BLOB etc…). I think FitNesse ‘represents’ everything as text. I think this works out ok for non-text types, but I haven’t tested it. In Java, when fetching different types of data from the database, the type is something you carefully have to pay attention to.

So I decided to embed SQL statements in the Java layer (the Java fixtures) that is beneath FitNesse. However, here the maintainability problem remains. Sure, the testing of queries is made much easier by using jUnit. Connections can definitely be managed better in the Java layer. And Java has tools to handle exceptions programmatically. But you still run the risk of scattering SQL queries in all shapes and sizes all across the code.

This last observation eventually led me on the path to the Java Persistence API, about which I will write in a follow-up blog post.