Click here to monitor SSC

The Making of SQL Response : Battle of the Testers

Published 12 September 2008 5:54 am

I’ve just closed my last bug on SQL Response (or at least the last one that’s left on my plate for this version). Code freeze is tomorrow after which it gets a complete going over by the testers and pending any major dramas should be released in the next few weeks.

 

When I joined Red Gate they said they tended to do short projects – a few months or so before shipping the new version of the product. Great! This was going to be a sharp contrast to my previous employment where we worked on essentially one large product (parts of which were 15-20 years old).  Projects there tended to be on the long side, but Response has been a 14 month project which is quite a while for Red Gate.

 

So the length of project wasn’t new to me, however the major contrast is that the team I’ve been on were responsible for the whole of the product and not just one little bit of it.  This means my sense of involvement and responsibility has been much greater. I’m also one of the people who have been on this project fulltime since the outset which is probably why I think of it more fondly. It’s my first major piece of work at Red Gate and as such it’s also shown me a very new and, in my opinion, a much better way to build software. Which is why I’ve really (really, really) enjoyed working here so far.

 

So what was new (to me) about the process? Well in terms of the overall structure –Design, Implement, Test, rinse and repeat – not much. It’s more the depth and execution of each of those steps

 

Design

Initially it started with our designer (Dom), the product manager (Colin), the initial project manager (Dan A) and myself going out and talking to customers and trying to build up a picture of what they wanted. 

 

For a start the very idea of having a user experience designer was a whole new one, in the past I had to come up with my own UI and given art is the probably the only exam I’ve ever failed you can rest assured that my aesthetic ability is fairly minimal.  The best way to describe it is that Dom designs the “User Experience” and it’s very much this design that drives the way we build the product. What does the user want to see, what do they want to do when they see it.  Obviously you can’t just throw this together – there’s some sketching and ideas and research about what we can/can’t do. I dug up the fruits of our very first ideas meeting.

 

Dom doesn’t write code – instead he mocks things up in Visio (he doesn’t even feel the need to be constrained by Visual Studio’s UI designer!). Every dialog is constructed, discussed as to what is actually possible and can be implemented before the heat death of the universe and refined.  Then we have to actually cut some code.

 

Implement

At first it was Dan and myself on the coding. Dan was also project manager but he left Red Gate sometime after our initial beta having helped with the re-design. David then came aboard and took over from Dan as project manager/coder and also Daniel joined as well (a different Dan/Daniel obviously!).  As far as coding goes that’s pretty much of muchness I guess – it’s what all software companies do which is why there is so much software out there. The problem is that I think a lot of places just do the cutting code and ship the result, or pay merely lip service to the other parts.  Here, however, Dom remained involved throughout as new issues came out to tweak or even redesign parts and to make sure each dialog was just the right shade of pink.

 

Here also there was another stark difference to previous jobs.  Our technical author (Brian) was already involved.  Rather than being asked to just knock up some documentation at the end of the release cycle he was already going over the designs and early builds making sure that what we wrote was not only English (another of my failings) but hopefully understandable by users rather than just by the developers. We have terminology documents to make sure the language is consistent and pretty much any text in the product has come from Brian. This means he also has a good understanding of the product and is able to put the interactive help in the right places and write actually helpful online documentation.

 

Testing

However… by far the biggest change and for me the best one is the testing.  Almost as soon as the first code hit the build server and a build came out, Jason was all over it.  

To get this kind of feedback so early was a revelation as it means that someone is essentially looking over your shoulder and pointing out your errors straight away. The benefit of this is enormous because the earlier on you can find the issue, the earlier you can fix it and the easier it is to modify the structure and design of code if necessary.  This is far better then finding it several months later when you discover that one thing you either didn’t realise or plain forgot about means you have to rip out and refactor your code. Of course by that point there’s no time or it’s just too painful to do that so you hack round the problem. Catching the issues early is so much less painful.

 

After a month or two Priya also joined in testing and then they would start to tag-team you. One would open a bug, you’d “fix” it and then the other would come back with some extra small detail and you had to “fix” it again

 

Obviously when coding you try to think of edge cases early on and plan to handle them, you might even write test cases to make sure.  However it’s still you thinking about the same problem.  The testers aren’t you and they come at things with a different perspective.  They can see a whole world of cases you don’t think about, they have access to several versions of Windows and SQL server and they use every synapse to bring your code to its knees.

 

It’s painful but you know it makes sense and it changes the way you code. You start to think not only how you would usually, but also how to outwit the testers, how will they test this and how can I deny them opening an issue.  You don’t just think of corner cases but you make sure you handle them right now and do the bit of extra work just to avoid seeing the look of glee of the their face as their mouse heads towards the “Create new issue” button.

 

Ultimately it’s a battle you will lose because there’s always something they can find.  You can at least try to make it hard and laugh at their desperation when all they have left is to ask you to add a period character to some text or just align this label 1 pixel to the left but you still fix them because, although it hurts, on the inside you know they are right.

 

Jason and Priya have to put up with a lot, from me at least. I joke that my first response to any bug is simply to close it and see how many actually come back (sometimes it’s even true). Sometimes I’ll close it saying “that’s as designed – there’s no way we can change that” and I know that if they call me over to “discuss” it that I’ll end up fixing it somehow because they’ll think of a way it can be done. I’ll argue anyway because at least it distracts them from finding other issues and you can’t let them have it to easy. 

 

So whilst this battle continues with thrust and parry, ultimately you’re on the same side trying to write a product that people actually want and like to use (which is probably why the tester nearly always wins).  The team works with testers and developers in a symbiosis with new features being tested early and retested often for regressions.  I’ve said to Jason that I couldn’t do his job and he made the same claim about mine, but working closely together we hopefully get the best out of both of use and hopefully that reflects in the product.

 

Thanks to all the team.

 

 

Clockwise from back left: David (project manager/developer), Claire (marketing), Dom (designer), Helen (manager), Colin (product manager), Brian (technical author), Daniel (developer), Nigel (developer), Priya (tester), Eddie (support champion).  Not in the photo: Jason (tester) – he’s afraid making images of him will divest him of his supreme ability to find bugs or something.

 

So I hope you get to try out SQL Response and I hope you like it.  There is a release candidate out right now and a survey you fill in to win a $100 Amazon voucher.  I’ll be heading for a beer or several very soon.

3 Responses to “The Making of SQL Response : Battle of the Testers”

  1. RobertChipperfield says:

    Awesome stuff… I still remember when I first started at Red Gate, the shock that someone could find just so much wrong with what was apparently a simple piece of code!

    What do you mean you tried documenting a database with a million tables and it took more than 5 minutes? Oh, OK, I’ll get the profiler out and make it faster…

  2. jason.cook says:

    It is certainly an eye-opener just how much can go wrong with trivial bits of code… it’s a scary concept that a couple of hundred lines of code could have a dozen of bugs in it.

    Something that has always amazed me is how many companies see development and testing as two separate parts of the cycle – you develop the code en masse, then you test it en masse. Both are evolutionary in nature and by feeding back any problems earlier in the process, it becomes easier (and cheaper) to resolve them.

    The downside of the evolutionary route is that it requires more skills from both sides – the developers require testing exposure and the testers require development exposure. I’ve seen testers at Red Gate debugging (and even fixing) race conditions and other complex problems in the code, and likewise I’ve seen developers doing performance and stress analysis and other test-related tasks… at the end of the day while we have our job roles, the 80/20 rule seems to apply (a developer is 80% coder and 20% tester, although Nigel would never admit it!)

    So Nigel “Won’t Fix” Morse (as he’s jokingly come to be known as in the team)… I take it the first round of drinks will be on you?

Leave a Reply