CliveT

Some notes on Reflector 7

2011-02-01T08:31:00Z

Both Bart and I have blogged about some of the changes that we (and other members of the team) have made to .NET Reflector for version 7, including the new tabbed browsing model, the inclusion of Jason Haley's PowerCommands add-in and some improvements to decompilation such as handling iterator blocks. The intention of this blog post is to cover all of the main new features in one place, and to describe the three new editions of .NET Reflector 7.

If you'd simply like to try out the latest version of the beta for yourself you can do so here.

Three new editions

.NET Reflector 7 will come in three new editions:

.NET Reflector
.NET Reflector VS
.NET Reflector VSPro

The first edition is just the standalone Windows application. The latter two editions include the Windows application, but also add the power of Reflector into Visual Studio so that you can save time switching tools and quickly get to the bottom of a debugging issue that involves third-party code.

Let's take a look at some of the new features in each edition.

Tabbed browsing

.NET Reflector now has a tabbed browsing model, in which the individual tabs have independent histories. You can open a new tab to view the selected object by using CTRL+CLICK. I've found this really useful when I'm investigating a particular piece of code but then want to focus on some other methods that I find along the way. For version 7, we wanted to implement the basic idea of tabs to see whether it is something that users will find helpful. If it is something that enhances productivity, we will add more tab-based features in a future version.

PowerCommands add-in

We have also included Jason Haley's PowerCommands add-in as part of version 7. This add-in provides a number of useful commands, including support for opening .xap files and extracting the constituent assemblies, and a query editor that allows C# queries to be written and executed against the Reflector object model .

All of the PowerCommands features can be turned on from the options menu. We will be really interested to see what people are finding useful for further integration into the main tool in the future.

My personal favourite part of the PowerCommands add-in is the query editor. You can set up as many of your own queries as you like, but we provide 25 to get you started. These do useful things like listing all extension methods in a given assembly, and displaying other lower-level information, such as the number of times that a given method uses the box IL instruction.

These queries can be extracted and then executed from the 'Run Query' context menu within the assembly explorer.

Moreover, the queries can be loaded, modified, and saved using the built-in editor, allowing very specific user customization and sharing of queries.

The PowerCommands add-in contains many other useful utilities. For example, you can open an item using an external application, work with enumeration bit flags, or generate assembly binding redirect files. You can see Bart's earlier post for a more complete list.

.NET Reflector VS

.NET Reflector VS adds a brand new Reflector object browser into Visual Studio to save you time opening .NET Reflector separately and browsing for an object. A 'Decompile and Explore' option is also added to the context menu of references in the Solution Explorer, so you don't need to leave Visual Studio to look through decompiled code. We've also added some simple navigation features to allow you to move through the decompiled code as quickly and easily as you can in .NET Reflector.

When this is selected, the add-in decompiles the given assembly,

Once the decompilation has finished, a clone of the Reflector assembly explorer can be used inside Visual Studio.

When Reflector generates the source code, it records the location information. You can therefore navigate from the source file to other decompiled source using the 'Go To Definition' context menu item.

This then takes you to the definition in another decompiled assembly.

.NET Reflector VSPro

.NET Reflector VSPro builds on the features in .NET Reflector VS to add the ability to debug any source code you decompile. When you decompile with .NET Reflector VSPro, a matching .pdb is generated, so you can use Visual Studio to debug the source code as if it were part of the project. You can now use all the standard debugging techniques that you are used to in the Visual Studio debugger, and step through decompiled code as if it were your own.

Again, you can select assemblies for decompilation.

They are then decompiled.

And then you can debug as if they were one of your own source code files.

The future of .NET Reflector

As I have mentioned throughout this post, most of the new features in version 7 are exploratory steps and we will be watching feedback closely. Although we don't want to speculate now about any other new features or bugs that will or won't be fixed in the next few versions of .NET Reflector, Bart has mentioned in a previous post that there are lots of improvements we intend to make. We plan to do this with great care and without taking anything away from the simplicity of the core product.

User experience is something that we pride ourselves on at Red Gate, and it is clear that Reflector is still a long way off our usual standards. We plan for the next few versions of Reflector to be worked on by some of our top usability specialists who have been involved with our other market-leading products such as the ANTS Profilers and SQL Compare. I re-iterate the need for the really great simple mode in .NET Reflector to remain intact regardless of any other improvements we are planning to make.

I really hope that you enjoy using some of the new features in version 7 and that Reflector continues to be your favourite .NET development tool for a long time to come.

We've completed the first iteration

2010-12-16T00:00:00Z

There are a lot of features in C# that are implemented by the compiler and not by the underlying platform.

One such feature is a lambda expression. Since local variables cannot be accessed once the current method activation finishes, the compiler has to go out of its way to generate a new class which acts as a home for any variable whose lifetime needs to be extended past the activation of the procedure.

Take the following example:

Random generator = new Random();
Func func = () => generator.Next(10);

In this case, the compiler generates a new class called <>c_DisplayClass1 which is marked with the CompilerGenerated attribute.

[CompilerGenerated]
private sealed class <>c__DisplayClass1
{
// Fields
public Random generator;

    // Methods
    public int b__0()
    {
        return this.generator.Next(10);
    }
}

Two quick comments on this:

(i) A display was the means that compilers for languages like Algol recorded the various lexical contours of the nested procedure activations on the stack. I imagine that this is what has led to the name.

(ii) It is a shame that the same attribute is used to mark all compiler generated classes as it makes it hard to figure out what they are being used for. Indeed, you could imagine optimisations that the runtime could perform if it knew that classes corresponded to certain high level concepts.

We can see that the local variable generator has been turned into a field in the class, and the body of the lambda expression has been turned into a method of the new class. The code that builds the Func object simply constructs an instance of this class and initialises the fields to their initial values.

    <>c__DisplayClass1 class2 = new <>c__DisplayClass1();
    class2.generator = new Random();
    Func func = new Func(class2.b__0);

Reflector already contains code to spot this pattern of code and reproduce the form containing the lambda expression, so this is example is correctly decompiled.

The use of compiler generated code is even more spectacular in the case of iterators. C# introduced the idea of a method that could automatically store its state between calls, so that it can pick up where it left off. The code can express the logical flow with yield return and yield break denoting places where the method should return a particular value and be prepared to resume.

        {
            yield return 1;
            yield return 2;
            yield return 3;
        }

Of course, there was already a .NET pattern for expressing the idea of returning a sequence of values with the computation proceeding lazily (in the sense that the work for the next value is executed on demand). This is expressed by the IEnumerable interface with its Current property for fetching the current value and the MoveNext method for forcing the computation of the next value. The sequence is terminated when this method returns false.

The C# compiler links these two ideas together so that an IEnumerator returning method using the yield keyword causes the compiler to produce the implementation of an Iterator.

Take the following piece of code.

        IEnumerable GetItems()
        {
            yield return 1;
            yield return 2;
            yield return 3;
        }

The compiler implements this by defining a new class that implements a state machine. This has an integer state that records which yield point we should go to if we are resumed. It also has a field that records the Current value of the enumerator and a field for recording the thread. This latter value is used for optimising the creation of iterator instances.

[CompilerGenerated]
private sealed class d__0 : IEnumerable, IEnumerable, IEnumerator, IEnumerator, IDisposable
{
    // Fields
    private int <>1__state;
    private int <>2__current;
    public Program <>4__this;
    private int <>l__initialThreadId;

The body gets converted into the code to construct and initialize this new class.

private IEnumerable GetItems()
{
    d__0 d__ = new d__0(-2);
    d__.<>4__this = this;
    return d__;
}

When the class is constructed we set the state, which was passed through as -2 and the current thread.

public d__0(int <>1__state)
{
this.<>1__state = <>1__state;
this.<>l__initialThreadId = Thread.CurrentThread.ManagedThreadId;
}

The state needs to be set to 0 to represent a valid enumerator and this is done in the GetEnumerator method which optimises for the usual case where the returned enumerator is only used once.

IEnumerator IEnumerable.GetEnumerator()
{
    if ((Thread.CurrentThread.ManagedThreadId == this.<>l__initialThreadId)
              && (this.<>1__state == -2))
    {
        this.<>1__state = 0;
        return this;
    }

The state machine itself is implemented inside the MoveNext method.

private bool MoveNext()
{
    switch (this.<>1__state)
    {
        case 0:
            this.<>1__state = -1;
            this.<>2__current = 1;
            this.<>1__state = 1;
            return true;

        case 1:
            this.<>1__state = -1;
            this.<>2__current = 2;
            this.<>1__state = 2;
            return true;

        case 2:
            this.<>1__state = -1;
            this.<>2__current = 3;
            this.<>1__state = 3;
            return true;

        case 3:
            this.<>1__state = -1;
            break;
    }
    return false;
}

At each stage, the current value of the state is used to determine how far we got, and then we generate the next value which we return after recording the next state. Finally we return false from the MoveNext to signify the end of the sequence.

Of course, that example was really simple. The original method body didn't have any local variables. Any local variables need to live between the calls to MoveNext and so they need to be transformed into fields in much the same way that we did in the case of the lambda expression. More complicated MoveNext methods are required to deal with resources that need to be disposed when the iterator finishes, and sometimes the compiler uses a temporary variable to hold the return value.

Why all of this explanation?

We've implemented the de-compilation of iterators in the current EAP version of Reflector (7).

This contrasts with previous version where all you could do was look at the MoveNext method and try to figure out the control flow.

There's a fair amount of things we have to do. We have to spot the use of a CompilerGenerated class which implements the Enumerator pattern. We need to go to the class and figure out the fields corresponding to the local variables. We then need to go to the MoveNext method and try to break it into the various possible states and spot the state transitions. We can then take these pieces and put them back together into an object model that uses yield return to show the transition points. After that Reflector can carry on optimising using its usual optimisations.

The pattern matching is currently a little too sensitive to changes in the code generation, and we only do a limited analysis of the MoveNext method to determine use of the compiler generated fields. In some ways, it is a pity that iterators are compiled away and there is no metadata that reflects the original intent. Without it, we are always going to dependent on our knowledge of the compiler's implementation. For example, we have noticed that the Async CTP changes the way that iterators are code generated, so we'll have to do some more work to support that.

However, with that warning in place, we seem to do a reasonable job of decompiling the iterators that are built into the framework. Hopefully, the EAP will give us a chance to find examples where we don't spot the pattern correctly or regenerate the wrong code, and we can improve things.

Please give it a go, and report any problems.

The Joy of Clojure (book review)

2010-09-30T15:02:35Z

It's been a long time since Lisp and Scheme have been in the news. Despite the abilities of these languages to enable fast prototyping of complicated applications, they got overlooked in the old days due to people's suspicion of garbage collection and their rather strange syntax.

In more modern times, the take up of virtual machine technology and associated technologies such as garbage collection has made it possible for some of these older languages to re-emerge. Indeed, not a day goes by without another language appearing which targets the Java Virtual Machine (JVM). One such language is Clojure, which is making quite a splash. Clojure takes some of the ideas behind Lisp and Scheme, mixes in some other ideas from functional programming, and blends this together with a large dose of practicality. The result is a language that runs nicely on top of the JVM and which allows the programmer to interoperate very easily with existing Java libraries and frameworks.

No more does the Lisp system have to be in charge of the process. Clojure code compiles down to Java class files which can loaded and exercised by a non-Clojure controller. Of course, things are not completely rosy. When you are debugging mixed- language applications, you get to see what the datatypes really look like and see more of the implementation details than you'd really like. However, we are now at a stage where different parts of a problem can be solved by a language which suits that particular problem area, and then these parts can be linked to form a coherent whole.

Lisp has had some notable successes in the past. Paul Graham's ViaWeb (which was bought by Yahoo! in the early days of the web) is one example, and, more recently, some of the travel scheduling companies have used this language to good effect. Clojure might just be one way of getting these advantages into the hands of the programmer-in-the-office. Lisp is very extensible and Clojure's creator, Rich Hickey, has used this to add many interesting concurrency constructs into the language. Since multi-core concurrency is the new hot topic, this makes the language even more deserving of consideration by the average programmer.

The Joy of Clojure is a book which tries to show experienced programmers how Clojure can solve some of the common problems that they have in their day jobs. Section 1.3 of the introduction neatly decomposes the problems with many other languages into the categories: verbosity, unavoidable boiler-plate, a long thought-code-feedback loop, incidental complexity, difficulties in extension, and deficiencies in supporting crucial programming paradigms. The book does a good job at relating these aspects to the parts of Clojure that help with them, helping to communicate the "why?" instead of just the "how?"

The book starts quickly, and continues at quite a pace. It is full of lots of interesting observations about programming in general and references lots of material from lots of sources. Chapter one, for example, starts out with the three pillars of Clojure - simplicity, freedom to focus and empowerment, and then leaps into some fairly large examples of the language's extensibility and its support for encapsulation, by way of an embedded Domain Specific Language (DSL) for a SQL query and the implementation of a chessboard. This chapter also does a really good job of explaining the rationale for functional programming via the metaphor of a child's flip book, and also explains how Clojure can solve the Expression problem using protocols. The only trouble covering material at this pace is that we are left with lots of forward references to the following chapters. This happens frequently throughout the book, and it's always unclear whether it is worth jumping forwards to follow up on something, or whether we should wait to reach the material. Personally, I like this style, but I imagine some people might find it annoying. The book is definitely worth a couple of reads and on the second pass through the forward referencing isn't quite as annoying.

Chapter 2 of the book covers the basic datatypes and syntax of the Clojure language, starting with basic strings and integers, and then covering the constructed types of list, vector, map and set. The emphasis on the latter two types is what makes this language different from more mainstream languages. We then move on to defining functions including loops and recursion, and then cover very basic Java interoperability and how namespaces affect the visibility of definitions.

Chapters 3, 4 and 5 go back through the scalars and the composite datatypes in more detail, encouraging the reader to experiment at the REPL, (the read-eval-print-loop), which is the means by which Clojure allows the programmer to experiment with the definitions that they have just made. Programming can be a very exploratory activity, and this is the way Clojure makes this possible.

Sequences in Clojure are a nice unifying concept. Most of the datatypes can be viewed as sequences, and chapter 6 takes this to the next level by explaining laziness and how it allows very succinct definitions of things like the triangle numbers. In this case, rather than defining a function to return the nth triangle number, it is much easier to define a lazy sequence of such numbers and then use the standard sequence functions to select a sub-range. This chapter contains a large worked example of implementing a lazy version of quicksort. This example is fascinating, especially as we can generate the first element of the sorted result by asking for the first element, and letting the laziness infer the work that needs to be done.

Chapter 7 covers functional programming in more depth, concentrating on closures and higher order functions. This chapter finishes with an example implementation of the A*search algorithm. Examples like these are one of the strongest parts of the book. Most chapters have examples that were well chosen, being neither too complicated (which tends to obfuscate the point) nor too easy (which misses the chance to show language features in context).

Chapter 8 covers macros, and does a reasonable job of covering all of the related material, with an emphasis on code-as-data. Again there's a good example to illustrate the material.

Chapter 9 moves on to more details about namespaces, multi-methods and records and protocols. This was useful material which was well presented, though having such disparate topics in the same chapter felt a little strange.

Chapter 10 covers the interoperability story, showing how easy it is to call Java methods, and how easy it is to expose functionality written in Clojure to the outside world. I've used Clojure a fair amount in my spare time and certainly learned some things from this chapter.

Chapter 11, entitled Mutation, covers the stateful side of the Clojure story, covering the software transactional memory (via Refs), agents, atoms and vars, with some additional material on using standard Java locks. There's also coverage of the primitives for launching parallel tasks as well as futures and promises. Clojure has a big emphasis on stateless programming, but coordinated changes of application state are required to write anything but trivial applications.

Chapter 12 covers performance, detailing how you can aid the compiler with type hints, which allow it to avoid boxing primitive values too frequently. It also covers local mutation in order to improve performance by using transients, and has a good discussion of memorization.

The final chapter, chapter 13, makes some final points about the usefulness of Clojure. It discusses how easy it is to define embedded DSLs, and how some of traditional design patterns can be implemented without requiring a lot of scaffolding.

This is a good book, which explains very well the major parts of the Clojure language, and contrasts the Clojure style against that of more mainstream languages. With such a range of concepts and ideas, the material is fairly well organised, though some chapters don't really have a single theme running through them. In some cases, the author references other external material, and it is a pity that there isn't time to go into more depth in the text itself. However, at 416 pages, it's clear that there just isn't the space to do this.

The examples concentrate on the language itself. There is nothing to show you how to write database access code in Clojure or how to hook up to a web server, or even how to use a build tool such as Leiningen to package your Clojure code into a deployable application. By focusing purely on the language instead of its use, the book can go into more technical depth than some other texts.

The Joy of Clojure explains both the "how" and "why" of Clojure, as it promises it will in the introductory chapter, and it is well worth a read. If you haven't done any Lisp or Scheme before, you might be better off reading a more introductory text first, but I can't see any book explaining the essence of Clojure better than this one.

F# and the rose-tinted reflection

2010-05-20T10:08:03Z

We're already seeing increasing use of many cores on client desktops. It is a change that has been long predicted. It is not just a change in architecture, but our notions of efficiency in a program. No longer can we focus on the asymptotic complexity of an algorithm by counting the steps that a single core processor would take to execute it. Instead we'll soon be more concerned about the scalability of the algorithm and how well we can increase the performance as we increase the number of cores. This may even lead us to throw away our most efficient algorithms, and switch to less efficient algorithms that scale better. We might even be willing to waste cycles in order to speculatively execute at the algorithm rather than the hardware level.

State is the big headache in this parallel world. At the hardware level, main memory doesn't necessarily contain the definitive value corresponding to a particular address. An update to a location might still be held in a CPU's local cache and it might be some time before the value gets propagated. To get the latest value, and the notion of "latest" takes a lot of defining in this world of rapidly mutating state, the CPUs may well need to communicate to decide who has the definitive value of a particular address in order to avoid lost updates. At the user program level, this means programmers will need to lock objects before modifying them, or attempt to avoid the overhead of locking by understanding the memory models at a very deep level.

I think it's this need to avoid statefulness that has led to the recent resurgence of interest in functional languages. In the 1980s, functional languages started getting traction when research was carried out into how programs in such languages could be auto-parallelised. Sadly, the impracticality of some of the languages, the overheads of communication during this parallel execution, and rapid improvements in compiler technology on stock hardware meant that the functional languages fell by the wayside. The one thing that these languages were good at was getting rid of implicit state, and this single idea seems like a solution to the problems we are going to face in the coming years.

Whether these languages will catch on is hard to predict. The mindset for writing a program in a functional language is really very different from the way that object-oriented problem decomposition happens - one has to focus on the verbs instead of the nouns, which takes some getting used to.

There are a number of hybrid functional/object languages that have been becoming more popular in recent times. These half-way houses make it easy to use functional ideas for some parts of the program while still allowing access to the underlying object-focused platform without a great deal of impedance mismatch. One example is F# running on the CLR which, in Visual Studio 2010, has because a first class member of the pack. Inside Visual Studio 2010, the tooling for F# has improved to the point where it is easy to set breakpoints and watch values change while debugging at the source level.

In my opinion, it is the tooling support that will enable the widespread adoption of functional languages - without this support, people will put off any transition into the functional world for as long as they possibly can. Without tool support it will make it hard to learn these languages.

One tool that doesn't currently support F# is Reflector. The idea of decompiling IL to a functional language is daunting, but F# is potentially so important I couldn't dismiss the idea. As I'm currently developing Reflector 6.5, I thought it wise to take four days just to see how far I could get in doing so, even if it achieved little more than to be clearer on how much was possible, and how long it might take. You can read what happened here, and of the insights it gave us on ways to improve the tool.

Towards an F# .NET Reflector add-in

2010-05-20T10:04:00Z

When I had the opportunity to spent some time during Red Gate's recent "down tools" week on a project of my choice, the obvious project was an F# add-in for Reflector . To be honest, this was a bit of a misnomer as the amount of time in the designated week for coding was really less than three days, so it was always unlikely that very much progress would be made in such a small amount of time (and that certainly proved to be the case), but I did learn some things from the experiment.

Like lots of problems, one useful technique is to take examples, get them to work, and then generalise to get something that works across the board. Unfortunately, I didn't have enough time to do the last stage.

The obvious first step is to take a few function definitions, starting with the obvious hello world, moving on to a non-recursive function and finishing with the ubiquitous recursive Fibonacci function.

let rec printMessage message =
printfn message

let foo x =
(x + 1)

let rec fib x =
if (x >= 2) then (fib (x - 1) + fib (x - 2)) else 1

The major problem in decompiling these simple functions is that Reflector has an in-memory object model that is designed to support object-oriented languages. In particular it has a return statement that allows function bodies to finish early. I used some of the in-built functionality to take the IL and produce an in-memory object model for the language, but then needed to write a transformer to push the return statements to the top of the tree to make it easy to render the code into a functional language. This tree transform works in some scenarios, but not in others where we simply regenerate code that looks more like CPS style.

The next thing to get working was library level bindings of values where these values are calculated at runtime.

let x = [1 ; 2 ; 3 ; 4]

let y = List.map (fun x -> foo x) x

The way that this is translated into a set of classes for the underlying platform means that the code needs to follow references around, from the property exposing the calculated value to the class in which the code for generating the value is embedded.

One of the strongest selling points of functional languages is the algebraic datatypes, which allow definitions via standard mathematical-style inductive definitions across the union cases.

type Foo =
| Something of int
| Nothing

type 'a Foo2 =
| Something2 of 'a
| Nothing2

Such a definition is compiled into a number of classes for the cases of the union, which all inherit from a class representing the type itself. It wasn't too hard to get such a de-compilation happening in the cases I tried.

What did I learn from this?

Firstly, that there are various bits of functionality inside Reflector that it would be useful for us to allow add-in writers to access. In particular, there are various implementations of the Visitor pattern which implement algorithms such as calculating the number of references for particular variables, and which perform various substitutions which could be more generally useful to add-in writers. I hope to do something about this at some point in the future.

Secondly, when you transform a functional language into something that runs on top of an object-based platform, you lose some fidelity in the representation. The F# compiler leaves attributes in place so that tools can tell which classes represent classes from the source program and which are there for purposes of the implementation, allowing the decompiler to regenerate these constructs again. However, decompilation technology is a long way from being able to take unannotated IL and transform it into a program in a different language. For a simple function definition, like Fibonacci, I could write a simple static function and have it come out in F# as the same function, but it would be practically impossible to take a mass of class definitions and have a decompiler translate it automatically into an F# algebraic data type.

What have we got out of this?

Some data on the feasibility of implementing an F# decompiler inside Reflector, though it's hard at the moment to say how long this would take to do. The work we did is included the 6.5 EAP for Reflector that you can get from the EAP forum.

All things considered though, it was a useful way to gain more familiarity with the process of writing an add-in and understand difficulties other add-in authors might experience.

.Net Reflector 6.5 EAP now available

2010-03-26T06:29:00Z

With the release of CLR 4 being so close, we’ve been working hard on getting the new C# and VB language features implemented inside Reflector. The work isn’t complete yet, but we have some of the features working. Most importantly, there are going to be changes to the Reflector object model, and we though it would be useful for people to see the changes and have an opportunity to comment on them.

Before going any further, we should tell you what the EAP contains that’s different from the released version.

A number of bugs have been fixed, mainly bugs that were raised via the forum. This is slightly offset by the fact that this EAP hasn’t had a whole lot of testing and there may have been new bugs introduced during the development work we’ve been doing.

The C# language writer has been changed to display in and out co- and contra-variance markers on interfaces and delegates, and to display default values for optional parameters in method definitions. We also concisely display values passed by reference into COM calls. However, we do not change callsites to display calls using named parameters; this looks like hard work to get right.

The forthcoming version of the C# language introduces dynamic types and dynamic calls. The new version of Reflector should display a dynamic call rather than the generated C#:

  dynamic target = MyTestObject();
  target.Hello("Mum");

We have a few bugs in this area where we are not casting to dynamic when necessary. These have been fixed on a branch and should make their way into the next EAP.

To support the dynamic features, we’ve added the types IDynamicMethodReferenceExpression, IDynamicPropertyIndexerExpression, and IDynamicPropertyReferenceExpression to the object model. These types, based on the versions without “Dynamic” in the name, reflect the fact that we don’t have full information about the method that is going to be called, but only have its name (as a string). These interfaces are going to change – in an internal version, they have been extended to include information about which parameter positions use runtime types and which use compile time types. There’s also the interface, IDynamicVariableDeclaration, that can be used to determine if a particular variable is used at dynamic call sites as a target.

A couple of these language changes have also been added to the Visual Basic language writer.

The new features are exposed only when the optimization level is set to .NET 4. When the level is set this high, the other standard language writers will simply display a message to say that they do not handle such an optimization level. Reflector Pro now has 4.0 as an optional compilation target and we have done some work to get the pdb generation right for these new features.

The EAP version of Reflector no longer installs the add-in on startup. The first time you run the EAP, it displays the integration options dialog. You can use the checkboxes to select the versions of Visual Studio into which you want to install the EAP version. Note that you can only have one version of Reflector Pro installed in Visual Studio; if you install into a Visual Studio that has another version installed, the previous version will be removed.

Please try it out and send your feedback to the EAP forum.

Reflector Pro has now been released!

2010-02-15T04:56:00Z

After moving into the .NET division in May , and having a great time working on Reflector, I'm pleased to say that the results of that work are now available. Reflector Pro has now been released!

The old Reflector as you know and love it is still available free of charge, and as part of this project we've fixed a number of bugs in the de-compilation that have been around for a long time. The Pro version comes as an add-in for Visual Studio - this offers dynamic de-compilation and generation of pdb files which allow you to step into the de-compiled code. Alex has some good pictures of this functionality on his beta post from around a month ago.

Thanks to the other guys who've worked on this for taking me along for the ride - Alex, Andrew, Bart and Jason. Stephen did some great usability work, Chris Alford did some great technical authoring and Laila handled the launch publicity.

Like all projects, there's always more I'd like to have done, but what we have looks like a pretty powerful addition to the developer's set of tools to me. Please try it and give us feedback on the forum.

Real-World Functional Programming (book review)

2010-01-03T08:39:00Z

I had been reading Tomas Petricek's blog for some time when he announced that he was going to be writing a book. I can’t remember when this announcement was made, and somehow the book seems to have taken a long time to come out, but Real-World Functional Programming (co-authored with Jon Skeet) was certainly worth the wait.

This book isn’t just a simple introduction to programming in F#; it’s an introductory text on functional programming covering the many reasons why it is time for this programming paradigm to finally be accepted by mainstream programmers. And it also contains much more...

The authors promote the idea of functional programming as a new way of solving problems, where ideas such as immutability and functions as values can be used to solve parts of a problem in data- and behaviour-centric ways. Two benefits of these ideas are that the concurrency may well become easier to extract from the solution, and it may be easier to unit test the resulting code. These are two issues that are currently receiving much debate in the programming community.

The functional examples are written in both F# and C#, and show how many of the features introduced in version 3 of C# allow a more declarative style of programming. The examples clearly demonstrate how many of the benefits of functional programming can be grasped by C# programmers. In particular, the book relates the use of LINQ in C# with the use of pipelining in F. Many of the examples show the brevity of the F# syntax which makes it the more attractive option for functional programming. The authors also demonstrate how easy it is to mix code in C# and F#, showing that it is may now be possible to write algorithm-centric code in F#, and more object-oriented code such as GUI code in C#. Along the way, there are some tricks for getting generics to behave in C#, and an interesting section that compares the type checking algorithms of the two languages.

The book doesn’t cover all of the syntax of F#, but describes the major parts. Indeed, the coverage of computational workflows, also known as monads, is probably the best that I have ever read. Later chapters of the book look at functional reactive programming, an interesting if somewhat esoteric theory that neatly matches the abilities of functional programming languages to express domain specific ideas with a declarative means of specifying graphical models.

There are also chapters on asynchronous and parallel programming, which emphasise the benefits of the functional style; the former contains a very interesting example of fetching data from a web service in order to visualise it via graphs. The chapter on reactive functional programming is preceded by one showing how easy it is to write compositional functional libraries. These chapters truly emphasise the benefits of the functional approach.

The book is fairly long at close to 600 pages, and in my opinion, there isn’t much that is wrong with it. The authors introduce the concepts layer by layer, with several sections telling you that more detail will be available in later chapters. Having done a fair amount of functional programming in the past, I found this stratum by stratum approach a little annoying to start with, but with every section containing lots of interesting observations there was little opportunity for skipping parts of the book. In the end, the richness of the observations outweighed my annoyance at the repetition. It would have been nice to have more discussion of functional data types, though there is a lot of material available elsewhere on this subject. The book also contained my pet bug-bear – the use of memorization to improve the performance of functional code without any discussion of how to avoid the memorization cache becoming a memory leak.

Overall, this is an amazingly good book that puts into words the many benefits of the functional paradigm, and shows that the F# programming language can embrace both the functional and object-oriented styles of programming. C# programmers, too, can gain something from this book, using it to understand the ideas behind LINQ and lambda expressions and why these have been incorporated into the language. I’m going to be re-reading it and working through the examples in more depth in the near future.

What a Job Opportunity...

2009-07-20T06:05:00Z

Life as a developer at Red Gate has many opportunities! One moment you’re finishing off code in the first release of a new tool and then you move into the .NET division to work with a group of talented but scary-looking individuals. What do you get to work on? A tool for allowing you to debug de-compiled source code in other assemblies than your own, from within Visual Studio.

As a developer there’s a lot to understand when you use the .NET framework and third-party component libraries. You can read the documentation and experiment with API functions, but there’s nothing better for improving your understanding of them than seeing the code and following through as it does its work. Using this tool, you will get to see the code for handling particular corner cases and the many subtle points which are never covered in documentation and are hard to deduce from a static view of the code. For example, I’ve already looked at how ASP.NET queries are processed and how the Windows Forms library interacts with the Windows message loop. With a better understanding of the behaviour of the code, when you find problems in a library, you have a better chance of any workaround not just working in a couple of cases that you’ve had time to test, but all the time.

I’m convinced that programming is an exploratory activity, and that this is a great tool for helping with the exploration.

This will work by adapting the Reflector de-compilation technology and using it to produce a .pdb file that links this generated code to the original assembly. A .PDB file is used by Visual Studio, and stores debugging information about a .NET module such as a list of all symbols in a module with their addresses, the name of the file and the line on which the symbol was declared. Bart talked a little about this project before and Andrew Hunter, aka the master of the dark arts, has been working on it for a while as a background project.

There’s a long way to go, but we’ve started to push harder at this project. As a developer, it’s an incredibly liberating experience to be able to step through the .NET framework at long last. In the past, I too have stared into the haystack of source code using Reflector. Reflector is great, but to understand the behaviour of a system it would be even better if one could see the code paths that are commonly taken and typical values for variables in the various stack frames.

So in this project, there are two sets of tasks to do. Obviously the de-compilation to source and the production of the .pdb file is one of them; the second is to integrate the production into Visual Studio to make it easy for the developer to select the assemblies that they wish to step through.

We can start with a simple project that calls into the framework code.

Hitting F5 runs some code in an add-in before the application starts that scans the solution to find all of the assemblies that are referenced. These are presented to the user, who may choose those that should have a .pdb and source code generated for them. Here I have simply selected the mscorlib.dll

In case you haven’t guessed, this dialog is a placeholder until the proper design is put into place.

Pressing “ok”, dismisses the dialog and Visual Studio walks to the breakpoint that I cunningly preset earlier.

The add-in generated source code and produced a .pdb file. Visual Studio’s search path was also modified to make the .pdb file available, so when we press F11 (to step into), we find ourselves inside the de-compiled code for the .NET framework with the locals window showing the value of the local variable, which Reflector has chosen to name “value”.

Stepping into and onwards, we get further into the framework where we can look at more interesting locals.

We can even set breakpoints.

Given that Reflector supports Visual Basic, after some recent work by Alex, we can instead choose to see the de-compilation in Visual Basic.

At present, there’s even some support for working at the IL level.

Of course the debugging experience isn’t seamless. We probably won’t go to the trouble of scoping the locals, so that only live local variables are displayed, and it isn’t always possible for Visual Studio to calculate the variable values when you are stepping through optimised code. Also Edit and continue will not be supported. However you can change the variable values in the locals window and set the next statement so the user can change the code path.

How close is this to a real product? It is some way off yet, and there are still some unanswered questions as to how well we can generate a .pdb for optimised and obfuscated code. We’re making progress though, and the results are encouraging. Builds are popping out of the build server and unit tests are transitioning between red and green at quite a rate.

And I’m having a great time working on it too!