Jason Crease

SmartAssembly Error Reporting: with the bug-finding power of more than 20 testers

Jason Crease — Thu, 08 Sep 2011 13:22:34 GMT

At Red Gate, we use the SmartAssembly automated-error reporting system in the applications we sell. When an error or exception occurs 'in-the-wild', it is reported back to Red Gate HQ, along with details about the error (stack trace, deployment environment, local variables, and more). Using the 'SmartAssembly Sync for JIRA' system, these errors are then automatically filed as bugs in our JIRA bug-tracking system. I should point out that Sync-For-JIRA is rather clever - it adds stack traces, it doesn't file duplicates, etc.

We have been using this system now for about a year, and were recently surprised to discover that this system files more bugs than our top 20 testers put together. SmartAssembly actually finds about 15 times as many bugs as one of our more prolific testers.

66% of issues (bugs, enhancements, etc) raised in the past year come from SmartAssembly Error Reporting.

In the past year, we have learned (rather embarrassingly) of thousands of different errors in our products. In the past we would have had to rely on good-hearted customers sending us in limited amounts of reproductory data, but thanks to SmartAssembly Automated Error Reporting we can automatically get all the data we need to reproduce the bug.

However, the type of bugs found by this handy tool tends to be different: unhandled exceptions, crashes, and unexpected environments. On the other hand, 'flesh-and-blood' testers tend to find UI glitches, workflow problems, etc. So, SmartAssembly error-reporting definitely complements real testers, rather than replaces them.

New spreadsheet accompanying SmartAssembly 6.0 provides statistics for prioritizing bug fixes

Jason Crease — Fri, 18 Mar 2011 12:06:00 GMT

One problem developers face is how to prioritize the many voices providing input into software bugs. If there is something wrong with a function that is the darling of a particular user, he or she tends to want action - now!

The developer's dilemma is how to ascertain that the problem is major or minor, and when it should be addressed.

Now there is a new spreadsheet accompanying SmartAssembly that provides exactly that information in an objective manner. This might upset those used to getting their way by being the loudest or pushiest, but ultimately it will ensure that the biggest problems get the priority they deserve.

Here's how it works: Feature Usage Reporting (FUR) in SmartAssembly 6.0 provides a wealth of data about how your software is used by its end-users, but in the SmartAssembly UI the data isn't mined to its full extent. The new Excel spreadsheet for FUR extracts statistics from that data and presents them in easy-to-understand forms.

I developed the spreadsheet feature in Microsoft Excel, using a fair amount of VBA. The spreadsheet connects directly to the database which stores the feature-usage data, and shows a wide variety of statistics and tables extracted from that data. You want to know what percentage of users have used the 'Export as XML' button? No problem. How popular is v5.3 is compared to v5.1? There's graphs for that. You need to know whether you have more users in Russia or Brazil? There's a big pie chart for that.

I recently witnessed the spreadsheet in use here at Red Gate Software.

My bug is exposed as minor

While testing new features in .NET Reflector, I found a usability bug in the Refresh button and filed it in the Red Gate bug-tracking system. The bug was labelled "V.NEXT MINOR," which means it would be fixed in the next point release.

Although I'm a professional tester, I'm not much different than most software users when they discover a bug that affects them personally: I wanted it fixed immediately. There was an ulterior motive at play here, of course. I would get to see my colleagues put the spreadsheet to work.

The Reflector team loaded up the spreadsheet to view the feature-usage statistics that SmartAssembly collected for the refresh button.

The resulting statistics showed that only 8% of users have ever pressed the Refresh button, and only 2.6% of sessions involve pressing the button. When Refresh is used, it's only pressed on average 1.6 times a session, with a maximum of 8 times during a session. This was in stark contrast to what I was doing as a conscientious tester: pressing it dozens of times per session.

The spreadsheet provides evidence that my bug was a minor one.

On to more serious things

Based on the solid evidence uncovered by the spreadsheet, the Reflector team concluded that my experience does not represent that of the vast majority of Reflector's recorded users. The Reflector team had ample data to send me back to my desk and keep the bug classified as "V.NEXT MINOR." The team then went back to fixing more serious bugs.

If I'm in the shoes of the user, I might not be thoroughly happy, but I cannot deny that the evidence clearly placed me in a very small minority. Next time I'm hoping the spreadsheet will prove that my bug is more important.

Find out more about Feature-Usage Reporting here.

The spreadsheet is available for free download here.

Breaking through the class sealing

Jason Crease — Mon, 26 Apr 2010 13:58:00 GMT

Do you understand 'sealing' in C#? Somewhat? Anyway, here's the lowdown.

I've done this article from a C# perspective, but I've occasionally referenced .NET when appropriate.

What is sealing a class?

By sealing a class in C#, you ensure that you ensure that no class can be derived from that class. You do this by simply adding the word 'sealed' to a class definition: public sealed class Dog {}

Now writing something like " public sealed class Hamster: Dog {} " you'll get a compile error like this:

'Hamster: cannot derive from sealed type 'Dog'

If you look in an IL disassembler, you'll see a definition like this:

.class public auto ansi sealed beforefieldinit Dog extends [mscorlib]System.Object

Note the addition of the word 'sealed'.

What about sealing methods?

You can also seal overriding methods. By adding the word 'sealed', you ensure that the method cannot be overridden in a derived class. Consider the following code:

public class Dog : Mammal { public sealed override void Go() { } }
public class Mammal { public virtual void Go() { } }

In this code, the method 'Go' in Dog is sealed. It cannot be overridden in a subclass. Writing this would cause a compile error:

public class Dachshund : Dog { public override void Go() { } }

However, we can 'new' a method with the same name. This is essentially a new method; distinct from the 'Go' in the subclass:

public class Terrier : Dog { public new void Go() { } }

Sealing properties?

You can also seal seal properties. You add 'sealed' to the property definition, like so:

public sealed override string Name
{
get { return m_Name; }
set { m_Name = value; }
}

In C#, you can only seal a property, not the underlying setters/getters. This is because C# offers no override syntax for setters or getters. However, in underlying IL you seal the setter and getter methods individually - a property is just metadata.

Why bother sealing?

There are a few traditional reasons to seal:

Invariance. Other people may want to derive from your class, even though your implementation may make successful derivation near-impossible. There may be twisted, hacky logic that could never be second-guessed by another developer. By sealing your class, you're protecting them from wasting their time. The CLR team has sealed most of the framework classes, and I assume they did this for this reason.
Security. By deriving from your type, an attacker may gain access to functionality that enables him to hack your system. I consider this a very weak security precaution.
Speed. If a class is sealed, then .NET doesn't need to consult the virtual-function-call table to find the actual type, since it knows that no derived type can exist. Therefore, it could emit a 'call' instead of 'callvirt' or at least optimise the machine code, thus producing a performance benefit. But I've done trials, and have been unable to demonstrate this If you have an example, please share!

All in all, I'm not convinced that sealing is interesting or important. Anyway, moving-on...

What is automatically sealed?

Value types and structs. If they were not always sealed, all sorts of things would go wrong. For instance, structs are laid-out inline within a class. But what if you assigned a substruct to a struct field of that class? There may be too many fields to fit.
Static classes. Static classes exist in C# but not .NET. The C# compiler compiles a static class into an 'abstract sealed' class. So static classes are already sealed in C#.
Enumerations. The CLR does not track the types of enumerations - it treats them as simple value types. Hence, polymorphism would not work.

What cannot be sealed?

Interfaces. Interfaces exist to be implemented, so sealing to prevent implementation is dumb. But what if you could prevent interfaces from being extended (i.e. ban declarations like "public interface IMyInterface : ISealedInterface")? There is no good reason to seal an interface like this. Sealing finalizes behaviour, but interfaces have no intrinsic behaviour to finalize
Abstract classes. In IL you can create an abstract sealed class. But C# syntax for this already exists - declaring a class as a 'static', so it forces you to declare it as such.
Non-override methods. If a method isn't declared as override it cannot be overridden, so sealing would make no difference. Note this is stated from a C# perspective - the words are opposite in IL. In IL, you have four choices in total: no declaration (which actually seals the method), 'virtual' (called 'override' in C#), 'sealed virtual' ('sealed override' in C#) and 'newslot virtual' ('new virtual' or 'virtual' in C#, depending on whether the method already exists in a base class).
Methods that implement interface methods. Methods that implement an interface method must be virtual, so cannot be sealed.
Fields. A field cannot be overridden, only hidden (using the 'new' keyword in C#), so sealing would make no sense.

How big is a string in .NET?

Jason Crease — Fri, 16 Jan 2009 16:32:00 GMT

How big is a string in .NET?

Typically the size of an object is 8 bytes for the object header plus the sum of the fields. Consider this simple object:

    class ThreeFields
    {
        double d;
        object o;
        int i;
    }

The size of a ThreeFields object is 8 bytes (for header) + 8 bytes (for the double) + 4 bytes (for the object pointer) + 4 bytes (for the integer) = 24 bytes . But what about a string?

A string is composed of:

An 8-byte object header (4-byte SyncBlock and a 4-byte type descriptor)
An int32 field for the length of the string (this is returned by String.Length).
An int32 field for the number of chars in the character buffer.
The first character of the string in a System.Char.
The rest of the string in a character buffer, terminating with a null terminator char. If you don’t believe me about the null-terminator, have a poke around with Reflector. String.AppendInPlace() demonstrates this nicely. One reason this is done is to aid unmanaged interop. Otherwise, everytime you marshal a string you’d need to copy it and add the ‘\0’

So a string size is 18 + (2 * number of characters) bytes. (In reality, another 2 bytes is sometimes used for packing to ensure 32-bit alignment, but I’ll ignore that). 2 bytes is needed for each character, since .NET strings are UTF-16. Thus, a string like so:

String mySimpleString = "Hello";

Will be 18 + (2 * 5) = 28 bytes. Splendid. Now consider this snippet of code. It creates a String and a Stringbuilder with the same contents.

           String atCompile = "123456789";
           StringBuilder buildingMyString = new StringBuilder("123");
           buildingMyString.Append("456");
           buildingMyString.Append("789");
           String fromStringBuilder = buildingMyString.ToString();

This should create two strings: atCompile and fromStringBuilder, which both read “123456789”. How big are atCompile and fromStringBuilder? You might think:

18 + (2 * 9) = 36 bytes

But if you profile it in a memory profiler, like ANTS Profiler, and you’ll see this.

The String atCompile is 36 bytes, as expected. fromStringBuilder has the same contents, but is 50 bytes. Eh? What’s going on there?

This weird behaviour is down to how System.StringBuilder does a ToString(). Like most people, I believed it allocated a new string and then copied the contents of the StringBuilder in. In reality, it just returns a reference to the string underpinning the StringBuilder. StringBuilders work by using strings backed with char buffers increasing in size in powers of two. A string does not need to be backed by a char buffer matching the string length; ‘expansion room’ is permitted. In this case, fromStringBuilder is backed by a 16-byte array, so is 18 + (2*16) = 50 bytes, as observed.

But what happens if the StringBuilder is then edited? Doesn’t the String we just got from ToString() then become invalid? Yes it does. When you do this append, StringBuilder copies the existing contents to a new string, and uses this new string. The String we got via ToString() continues to point to the String that has been discarded by the StringBuilder. This String is still backed by an over-sized char[].

I presume Microsoft made this design decision because it is a common idiom to create a StringBuilder, append to it, ToString() it, and then never use it again. Copying all those bytes from the StringBuilder to a String would be a waste. Even if you then append to the StringBuilder after doing your ToString(), the resulting copy of the StringBuilder‘s underlying string requires no time than would copying it during the ToString().

Scrum: Keeping Kosher doesn’t make you Jewish

Jason Crease — Mon, 08 Dec 2008 16:10:00 GMT

Our development team has started using Scrum. We’re only 4 weeks in, so it would be premature to make a judgement on whether it's working for us. I’ve investigated how it’s practised elsewhere, and there seems to be a worrying trend…

Let’s say I want to become a Christian. I could go to church on Sundays; receive communion; marry in a Church; sing hymns; celebrate Christmas. But none of these would make me a Christian. These are just extrinsic procedures – not core beliefs. To really be a Christian, you have to ‘live-for-God’; believe in the divinity of Christ; and so on. The point of Christianity is these core beliefs, not the extraneous rituals.

When it comes to Scrum, I always see teams faithfully having daily stand-up meetings, having retrospective meetings, calling someone a Scrummaster, etc. But they fail to implement the core principles of Scrum: ‘Done-is-done’, etc. It’s easy to practise the irrelevant rituals, yet these should not be confused with what Scrum is really about.

Many people tell me that no one implements Scrum properly – it’s a matter of compromise and seeing what works. I don't accept this. It’s remarkable how readily people compromise on the core-beliefs, yet retain all the extraneous procedures. They give up believing Jesus is the son of God, but keep Easter.

This is not to say there isn’t value in some of Scrum. Daily stand-up meetings are useful; but this hardly amounts to a vindication of Scrum. In the same way, some hymns are great (‘Autumn Days’ and ‘Shine Jesus Shine’ are two of my favourites) but this is not sufficient to justify becoming a Christian. If you like hymns but don’t believe in the divinity of Christ, then don’t become a Christian. Start a choir instead.

Nullable Structs - An interesting 'Gotcha'

Jason Crease — Wed, 26 Nov 2008 17:11:00 GMT

One of the interesting new features in C# 2.0 was nullable valuetypes. Using these, you can set valuetypes to a value, or null. Their usage is entirely straightforward. For instance, to use a nullable int simply declare a variable of type int?, and then set it to a value or null.

Behind the scenes, nullable valuetypes are implemented as generic structs of type Nullable<T>. When you write:

int? i = 2;

What you’re really writing is:

Nullable<int> i = new Nullable<int>(2);

When you assign a value to a pre-existing nullable int, like so:

i = 4;

you’re really writing:

i = new Nullable<int>(4);

And when you ‘get’ the value, like so:

int j = i;

What you’re really writing is:

int j = i.Value;

The Value is a get property, and assignment is by creating a new instance of Nullable<T>. This is similar to strings in C#: presenting a reference type by valuetype semantics.

This is all fine for primitives like int, double, etc. But when using structs, an interesting problem is created that lets light in on the ‘magic’ of nullable valuetypes.

Let’s say I have a struct Point, implemented as so:

struct Point

{

public int x, y;

public Point(int px, int py)

{

x = px; y = py;

}

public override string ToString()

{

return "(" + x + ", " + y + ")";

}

Then, I write a program that creates an int and a Point, and alter them slightly:

static void Main(string[] args)

{

int i = 3;

Console.WriteLine(++i); //increment then print it

Point p = new Point(2, 4);

p.x = 1; Console.WriteLine(p); //move left then print it

Console.ReadLine();

}

No problems here. The int and the Point will be altered as expected. Now I decide that I want my int and Point to be nullable. So I simply add two question marks, producing this program:

static void Main(string[] args)

{

int? i = 3;

Console.WriteLine(++i); //increment then print it

Point? p = new Point(2, 4);

p.x = 1; Console.WriteLine(p); //move left then print it

Console.ReadLine();

}

But this won’t compile. The int? part is fine, but for the Point? part I get the error: “'System.Nullable< Point>' does not contain a definition for 'x'”. No problem, I’ll just edit it like so:

static void Main(string[] args)

{

int? i = 3;

Console.WriteLine(++i); //increment then print it

Point? p = new Point(2, 4);

p. Value.x = 1; Console.WriteLine(p); //move left then print it

Console.ReadLine();

}

Now you get a different compile error: “Cannot modify the return value of 'System.Nullable<.Point>.Value' because it is not a variable”. This happens because Nullable<T>.Value has no set accessor method.

Nullable value types have this ‘limitation’ for a very good reason. The same thing happens is you try to edit a struct in a List<> of structs. Consider an over-simplification of Nullable<T> with a bodged-in set accessor, like so:

class MyNullable<T>

{

private T val;

public T Value

{

set {

val = value;

}

get {

return val;

}

If I create a MyNullable<Point> and use it:

MyNullable<Point> p = new MyNullable<Point>();

p.Value = new Point(2, 1);

p.Value.x = 2;

I get a compile error on the third line: “Cannot modify the return value of ‘MyNullable<Point>.Value' because it is not a variable”. This is because the statement p.Value.x = 2 is equivilent to:

Point otherPoint = p.Value; otherPoint.x = 2;

Which doesn’t change the underlying p.val, because otherPoint is a copy of p.Val. So when C# gives us a compile error, it is merely protecting us from editing an irrelevent copy of the underlying Point.

In conclusion, nullable valuetypes are immutable. I think the C# designers intended programmers to never modify the value of a struct once one is created, nullable or not.

Testing the speed of ANTS Profiler 4

Jason Crease — Tue, 16 Sep 2008 13:55:00 GMT

Profiling and debugging code inevitably adds overhead, and I know that it can be really frustrating. The overhead can either be small, e.g. the Visual Studio debugger, or massive, as with most performance profilers. A developer may invest in a performance profiler to optimize an algorithm that takes 10 minutes to run, and finds that when profiled it takes 2 hours to run. A top priority in the development of ANTS Profiler 4 was to make it faster than its predecessor and faster than any other profiler on the market.

We have found ANTS Profiler 4 to be about eight times faster than its predecessor, ANTS Profiler 3. It is also 4 to 20 times faster at line-level profiling than other profilers on the market.

I did a few speed tests and I have made a video to show how ANTS Profiler 4 compares to the fastest-known other profiler:

ANTS Profiler 4 vs fastest known competitor

Here it is compared to ANTS Profiler 3:

ANTS Profiler 4 vs ANTS Profiler 3

If you want to try out ANTS Profiler 4, you can download it from the RedGate website

A Quick .NET Puzzle

Jason Crease — Thu, 17 Jul 2008 17:17:00 GMT

Just a quick .NET puzzle. Does this application ever throw that ApplicationException? If so, why?

using System; using System.Threading; class Program { static long Num = 0; static void Main(string[] args) { Thread t1 = new Thread(ModifyNum); t1.Start(); while (true) { long k = Num; if (k != -1 && k != 0) throw new ApplicationException( "k is not -1 or 0. It is " + k.ToString()); } } static void ModifyNum() { while (true) { if (Num == -1) Num = 0; else Num = -1; } } }

Stuck? Then here's a hint: if you change Num from a long to an int, it never throws the exception.

Order of Construction

Jason Crease — Mon, 02 Jun 2008 15:05:00 GMT

For me, inheritance is often a headache. In particular, in what order is everything initialized? Consider this short C# program. It creates an instance of Dog, which derives from Animal. Both Dog and Animal have an instance constructor, a class initializer (aka a static constructor), an instance variable, and a static variable.

using System;



namespace CtorOrder

{

    class Animal

    {

        public int instanceAnimal = Program.Print("Animal.instanceAnimal");

        public static int staticAnimal = Program.Print("Animal.staticAnimal");

        public Animal()

        {

            Console.WriteLine("Animal constructor");

        }

        static Animal()

        {

            Console.WriteLine("Animal class constructor");

        }

    }

    class Dog : Animal

    {

        public int instanceDog = Program.Print("Dog.instanceDog");

        public static int staticDog = Program.Print("Dog.staticDog");

 

        public Dog()

        {

            Console.WriteLine("Dog constructor");

        }

        static Dog()

        {

            Console.WriteLine("Dog class constructor");

        }

    }

 

    class Program

    {

        public static int Print(string s)

        {

            Console.WriteLine(s);

            return 0;

        }

        static void Main(string[] args)

        {

            Dog d = new Dog();

            Console.WriteLine("\nPress enter to continue");

            Console.ReadLine();

        }

    }

}

There are eight activities here:

Dog instance constructor
Dog class constructor
Dog instance variable
Dog static variable
Animal instance constructor
Animal class constructor
Animal instance variable
Animal static variable

The question is: ‘In what order are the eight activities run?’ Try working it out. Anyway, the answer is:

Dog static variable
Dog class constructor
Dog instance variable
Animal static variable
Animal class constructor
Animal instance variable
Animal instance constructor
Dog instance constructor

There are two things I find surprising about this:

Why do static constructors run from most derived class to least derived class, i.e. Dog then Animal?
Why do instance variables get initialized before calling the parent’s instance constructor?

The answer to question 1 is that Dog’s class initializer has to be run since we are running Dog’s instance constructor, but execution of any type's class initializer method will not trigger automatic execution of any class initializer methods defined by its base type. Indeed, why should it?

The answer to question 2 is complicated. Constructing instance fields in the most derived class first is the only way to ensure that readonly fields are initialized and then not changed. Check out Eric Lippert’s blog for a decent explanation: http://blogs.msdn.com/ericlippert/archive/2008/02/15/why-do-initializers-run-in-the-opposite-order-as-constructors-part-one.aspx

Using the Exception Hunter Command Line to find All Methods that throw a given Exception

Jason Crease — Thu, 31 Jan 2008 12:01:00 GMT

Red-Gate's new .NET tool - Exception Hunter - shows you the exceptions that can be thrown by a .NET method. However, many users would like to see this the other way round, i.e. pick an exception and see which methods throw that exception. We will endeavour to put this in the next version of Exception Hunter.

However, for the moment this functionality is available via the Exception Hunter command line. I've recorded a 5-minute video where I demonstrate how to use the command-line to find all methods that throw IO exceptions in a simple application. Have a look...

http://www.simple-talk.com/blogbits/jason_crease/Exception_Hunter_Command_Line_demo_skin.swf

To download the 14-day free-trial of the .NET Developer Bundle, which includes Exception Hunter, go to:

Download .NET Developer Bundle