As Exchange Server Archiver slowly but steadily heads towards launch, we have completed more and more components. Recently the Outlook Add-in for Outlook 2003 was finished and became available for testing. I’ve been assigned to test the search functionality as well as the search interfaces. At first, this sounded like an interesting and easy task to do and I didn’t really understand why we assigned a total amount of 11 working hours for these tasks during planning. However, it all became clear once I actually started investigating what was to be done.

First of all, I needed to summarize all the possibilities for searching in Outlook. There are three basic types of mails: HTML, RTF and plain text. They all have different types of actual mail bodies, so I clearly had to test them all. There are several folders (e.g. Inbox, Sent box, Drafts, Deleted Items etc. plus the various subfolders that might spawn from any of these, to... well, a yet undefined possible depth) not to mention the possibility of adding and handling multiple mailboxes from one single Outlook. We offer single search from the toolbar, single search using the Outlook Add-in search window and there is an advanced search as well – which (un)fortunately offers criteria for nearly all frequently used e-mail properties such as “From”, “To”, “Subject”, “Sent date”, “Received date”, “Folder”, “Contains attachment” and so on, plus any combination of any of these (it is possible to add four “Subject” fields, the criteria will be ORed instead of ANDed as it would be if the words were entered into the same field). Then I figured out that these are just the components that we provide, not the basic search functionality as such, which also needed testing.

At this point, I started to become desperate and asked my good friend Google for help. I was wondering if there was a well-known algorithm that I should follow to thoroughly test such a function.  I typed in “testing search” in the hope that I might get pointed to a useful page where someone had already collected the most common things to look at. I admit it was a pretty lame try. I got 247,000,000 results from this try; sticking other words into the query got me down to 47,000,000. Well, maybe it wasn’t such a good idea to consult the World Wise Web with this one.

After wasting some hours trying to find the best method for what I was about to do, I decided that I’ll finally take some action. I just wanted to get a little bit of the job done that day. Then I faced another problem. If I wanted my test data to be more or less realistic, I’d need a huge amount of mails, with mostly garbage in them, and occasionally a useful set of words that will be matched. Randomly generating strings was out of the question. We used lorem ipsum with our other tests before, so I decided to stick with that method, with occasional self-forged mails as well – when I wanted something specific to occur.

The next thing on the list was to decide the directory structure in which the mails would reside. A lot of people (including myself) sort their mails into different directories and their subdirectories, not to mention our folder search function (you can search based on the original folder name of the mail), so I needed something realistic here as well. Something that is slightly more than normal use, but still isn’t totally insane. After some intense rubbing of my forehead I created ten major folders, with some subfolders in half of them, one of them containing a four level deep sub. I then scattered our test mail set (consisting of about 150 mails) in these and started the search.

I was told that the search had been refined, and it should only come up with exact matches, unlike the indexing service which is so sweet and handy that if you search for e.g. “best” it doesn’t only match instances of the specified word, it also matches its synonyms. Well, our point of view is that if someone searches for “best” that has a reason. Probably because the person wants to get mails that have the word specified. If the user wants to see mails that contain “well” or “good”, then he shouldn’t specify “best” as the criterion. So after all the long puzzled hours I spent with trying to figure out how to do this properly, I was aching for some action. I cut out a piece of a lorem ipsum mail and searched for it. I expected 1 result. Rather surprisingly, the engine came back with 47, of which 1 contained the string I was looking for, but none of the others did (ok, I admit, I only checked around 10 of them, so let’s assume that none of the other 36 contained it either). I performed some other queries. They were all wrong, so I went to the magical developer (a.k.a Rob) to call for aid. He checked that I’m really not completely bonkers yet, and directed me to an other developer (also known as Richard) who made the changes in the engine. He fired up his virtual system, did what I did previously, and had exactly one result. Interesting. I filed a bug and left him to play with it. After about 20 minutes a mail arrived from him with “Hahahahaha” as subject. His storage service (the component that communicates with Windows Indexing Service and does the search) was running on Windows XP, whereas mine was on Vista. The Indexing service has a particular bug on Vista: if the query has “LIKE” in it, the results will be completely messed up. Fortunately there’s a hotfix for it. After installing that, I could go and explore further. Since that was the funniest bug I found, I really don’t want to bore anyone with more.
After I finished... Well, not really... After I managed to convince myself that I had tested everything that should be tested (numbers, strings, words next to punctuation marks and so on) I tried to be little bit evil (normally I’m not. Honest. And I’m not sarcastic either!). Rob drew my attention to Windows’s noise word list – the words that will be completely ignored by the indexing service, and therefore by us as well. The list is really quite long, and to my surprise it is possible to create a valid English sentence with only noise words in it. According to my information, a mail with only that string in it is basically unsearchable. This seemed like something fun to try, so I created a mail that contained only the text in question, one that had something else as well, and a third one that had something else as well but was in the deepest subfolder. The latter two were unfortunately fairly useless, since it turned out with the first one that we just ignore the complete search criteria. After trying to put the text between quotation marks, I got an even funnier result:

Exception performing search: Incorrect syntax near 'between'. Expected '&', '\'', ')', '|', '~', OR, AND, NEAR. SQLSTATE=42000.

“Between” is the first word of my magical string. After a bit of playing, I found that the error will actually specify the first word of the string as the source of the error, but only if the text is actually between quotation marks – if there is only one of them, it all works fine, and even matches the desired strings. I filed this bug as well and considered the tests finished. After a while Richard asked me to come and see that this problem was not reproducible on his system, so we actually suspect that this is going to be a Vista issue again, though it is still under investigation.