Software Debugged

Thanks to John for his comment that asks a couple of questions I should have addressed in my Methodology and Madness - Introducing Process to a Development Group article. John’s questions are:

• How do you decide which to solve first?
• How long do you leave between introducing each solution?

How Do You Decide Which to Solve First?

I think the best way is simply to solve the problem that looks like it will be simplest to solve: identify it and introduce a way of working, that is a practice, that solves it. Note that the practice may or may not be supported by a tool. When this is done, look around for the second simplest, and proceed like that. This assumes there is nothing that obviously needs immediate attention. For example, if there is a problem that is causing late delivery, or causing the software to be delivered in a substandard state, then that problem must be solved as a matter of urgency!

How long do you leave between introducing each solution?

I think the best approach here is to introduce a practice/tool, then wait until it has settled in and become part of the development group’s way of working. That is, when people start following that practice without having to make a concerted effort to do so. Then it is time to think about what to do next.

Finally

As more practices (and tools) get introduced into a development group, these practices will soon start to show an improvement that is more than just the improvement any individual practice offers. Soon the development group will find itself following a development process. An important thing to realise is that the process does not need to be optimal from the onset. The first objective is to get out of the mud and into some sort of sensible way of working. Once this is achieved, the group is in a position from which it can improve its process.

Going back a couple of years, the development group I was contracting in at the time had decided to move out of its present chaotic state and start using a development process. I wonder how well that sentence conveys what they were planning to do? Just to clarify, the plan was this: all permanent development staff would go home one leaving the chaos behind, and when they came back the following morning they would be following whatever process/methodology they adopted.

Oh well. I’ve heard of this sort of madness before, but that’s one of only two instances I’ve come across it in my personal experience. I’m sorry to say it, but I was actually involved in the other one! Don’t tell any one – it was years ago and I’ve learned a lot since then. In passing, note that I’m interested in hearing such experiences – or indeed any related experiences – from other people.

Anyway, when the contract client I alluded to above went methodological my contract was coming to a close anyway. Consequently I wasn’t around long enough to see how it turned out for them. Note that above I refer to their approach as madness. This is because I think the problem of this kind of taming of a development environment is too big to solve in one go – even if the development group is quite small, say only five people. My own experience was that when a blanket process is introduced it doesn’t last very long. People either just go back to the way they used to do things, or they just never really start using the process. Either way, the process fails.

How then, should a development group move from chaos to discipline? The answer is simple: solve on problem at a time! Look around and identify something that is not working well, and work out how to change it for the better. For example, a few years ago I was working on a project for which the code was all written in C++. We had a couple of programmers who, although strong on some of the project technology (such as obtaining telemetry from various hardware) were relatively inexperienced in C++ programming techniques. We addressed this problem by introducing pair programming, to be used as and when needed. The technical lead was adept at keeping closely in touch with team members anyway, and simply organised the pairings as and when necessary. Note that although my C++ is/was very strong, I had been working on the project for a much shorter time than these two programmers. Therefore I was almost always paired as the “C++ mentor”, because then I could learn other aspects of the project from the programmer I was working with – thus making the pairing an exchange from which both of us, and hence the project, benefited.

Many thanks to John for commenting on my Performance Tuning article. I’d just like to quote a snippet of what John has to say:

“After all there’s no point optimising code that doesn’t actually work correctly”

It reminded me of something obvious I forgot to mention in my previous articles: another of the “sins” I have seen often seen programmers committing is trying to get code to run fast at the same time as they first writing code. Also, these programmers seemed (usually) to get themselves into a tangle, which is hardly surprising, because they make things difficult for themselves by trying to solve two problems at once. They are trying to both get their code working (in an operational sense), and speed it up, both at the same time. This ties in with what I said in my Performance Requirements article: programmers trying to optimise code while they first write it are trying to solve a performance problem, while having no idea if this problem actually exists or not.

Therefore, the correct approach is to first get code working operationally by using the simplest implementation possible. This should be quite quick (relatively speaking) and must include the writing of test cases. Then, performance can be measured and, if necessary, the code speeded up. Naturally the test cases must be run against the changes, thus verifying that functionality has not been broken.

In my previous article on Performance Requirements I argued that performance issues are requirements, and that this is not well understood in the software development community in general. In this article I’m going look at the performance tuning of existing code, argue that this is also poorly understood, and discuss the right way to go about it.

My friend Alan Griffiths related an experience of his from a few years ago. He was working on a desktop product written in C++. In this case the performance requirements of the software had been managed sensibly, in that performance criteria were understood. Alan was tasked with improving the performance of specific functionality by a factor of two.

How to go about this? Well, the functionality he was tasked with optimising used did quite a lot of searching and sorting of data. This had led to an implicit assumption (among some members of the development group) that this is where the improvements would be needed. However, the C++ development environment in use had a profiler, so Alan set about obtaining concrete information on where the problem really was – and the result was something of a surprise! Execution seemed to be spending a lot of time in string::operator== ! A quick look in the library source code quickly gave a clue: the operator== implementation simply called the C strcmp function. Alan decided that he had, at this point, obtained enough information to try something simple: he wrote a replacement operator== that first checked if the strings were of equal length, and simply returned false if they were not. This was enough to solve the problem! In fact, it delivered a factor of four improvement – twice as much as was deemed necessary – at no extra charge!

I have, many times throughout my career, seen programmers dive in and start speeding up code based on guesswork and unfounded assumptions – indeed, in my less experienced days I dare say I was guilty of this myself (just don’t tell anyone). This is a waste of time. Alan could easily have spent lots of time trying to optimise searching and sorting code, when this time would have been wasted. Unless you actually measure the performance of various parts of the code, you don’t know what is causing the code to under-perform. As illustrated above, this is true even if there are “obvious” candidates!

The moral of this story is a recurring theme: avoid guessing! Instead, gather concrete evidence of where the problem lies before making changes to the code.

Performance – in this case meaning a piece of software’s run time speed – is one of those things that always seems to cause problems whenever and wherever it comes up. Unfortunately, I think this is because it is very poorly understood across the software development community in general.

Harking back to a contract I had many months ago, I was tasked with improving the performance of a key component. I asked how much improvement was needed, and although I can’t remember my manager’s exact words, the reply was along the lines of: “It needs to run as fast as possible”. That wasn’t the first time I’d heard such as response, and I very much doubt it will be the last. This is worrying because such statements are basically madness, for the following reasons:

• It indicates the lack of a plan, that the requirements have not been understood, and that people who should know better are basically making things up as they go along; further, it indicates that those responsible for the requirements did not understand that performance requirements are requirements!
• Further, it means developers tasked with squeezing more speed out of a component will possibly (probably?) waste their time doing so. There is no point in spending time trying to get a factor of five improvement, when a factor of two is all that is needed!

Note that I’m not saying performance requirements need to be specified up front before any development can be done. On the contrary, much modern software is developed using agile methods which are designed to allow for changes in requirements during the development lifecycle. However, the fact that requirements can change is not an excuse for not understanding requirements. It does not matter if are requirement has just been updated or has even just been introduced into the project – it must still be understood!

In conclusion, what I’m saying is this: performance requirements are requirements, they must be treated as such and understood before any action is taken.