Click here to monitor SSC

Adventures in Web Testing

Test Engineer at Red Gate Software Ltd.

Switching from Selenium 1.x to WebDriver/Selenium 2 and HtmlUnit

Published Thursday, August 26, 2010 7:34 AM

Recently I became aware of the work that is being done to merge the WebDriver and Selenium codebases. The result is Selenium 2, a project that aims to offer the best bits from both. For more details about the merge, see here.

Having only used Selenium and not WebDriver before, there were a couple of things about WebDriver that piqued my interest. The first is the very clean object-based API that it provides, allowing you to make calls such as:

browser.FindElement(By.Id("q")).SendKeys("Simple Talk");

to type into a text field (in this case the search box on the Google home page). I like the clarity that this syntactical style provides, making it easy to understand existing tests and write new ones.

The second thing was WebDriver's support for HtmlUnit. HtmlUnit is a Java project that implements a UI-less web browser entirely in memory. This makes the process of browser automation much more efficient, as the overhead of creating a new browser instance is greatly reduced. Unfortunately, since its a Java project, taking advantage of HtmlUnit from .NET hasn't been a simple prospect up until now (although Steve Sanderson wrote a great blog post about how to do it using IKVM, see here). With the advent of Selenium 2, this process gets a whole lot easier!

The Selenium 2 alpha builds can be found here. I fired up the standalone server in a virtual machine, and create a new WebDriver against that machine as follows:

new RemoteWebDriver(new Uri("http://testVM:4444/wd/hub"), DesiredCapabilities.HtmlUnit());

Note the URL - to create a DefaultSelenium against this instance of the server, the URL would be http://testVM:4444/, whereas to create a WebDriver it is necessary to tack the /wd/hub onto the end.

To compare the performance of using HtmlUnit against Firefox (as I was previously) I copied one of my test fixtures into a new project and migrated the 12 tests over to use a RemoteWebDriver instead of a DefaultSelenium. Since I last wrote about using Selenium Grid, the number of tests has grown from 372 to 2736. To deal with this growth I increased the number of machines in the Grid to 8, the net result of which was that a fixture containing 12 tests running in at most 8 concurrent threads would take about 1 minute 30 seconds to execute. By comparison to this, 12 WebDriver/HtmlUnit-flavour tests running serially (since the Selenium Grid project doesn't support WebDriver) took only 24 seconds! that's almost 4x quicker (3.75x, to be exact), despite the lack of parallelism.

This convinced me that it was worth switching all 2736 tests over to use WebDriver, in spite of Selenium Grid's lack of support for it. So after a couple of hours work migrating my tests and helper methods over to the new API, I was ready to do a full test run. Against the Grid (and its 8 machines), this took almost exactly 3 hours. Using HtmlUnit, the test run completed in 45 minutes for a 4x speedup :)

So even in these early stages, the Selenium 2 project is shaping up to be pretty exciting for .NET web testing. I look forward to getting my hands on an early build of Selenium Grid that supports WebDriver, so I can start taking advantage of HtmlUnit in multiple tests concurrently :D

Comments

 

RossPatterson said:

I'm curious - do you attribute the 75% reduction in execution time primarily to Webdriver or HTMLUnit?  My expectation is the latter, but I don't know for sure.
September 2, 2010 3:01 PM
 

Ben Adderson said:

@Ross - definitely the latter. The vast majority of the performance gain comes from how much faster it is to start up and drive an HtmlUnit instance than a Firefox instance. Even if WebDriver were less efficient than Selenium (which I doubt) the loss due to that is massively outweighed by the gain due to using HtmlUnit.
September 2, 2010 4:20 PM
 

bitknower said:

Hi there,

My company is seriously considering using Selenium to test our webapp, but we're trying to figure out if we should start with using SE1 and later migrate to SE2 or if we should just start out using SE2.  It's very important to us that we can test in IE, FF, Chrome and Safari.

1.) How easy do you think it would be to migrate tests from SE1 to SE2?  Your article mentioned that you migrated a couple thousand tests in just a few hours...does this mean that it's a fairly trivial task?  If so, we'll probably start off using SE1 and move to SE2 later.  If not, then we might use SE2 right away.

2.) I noticed that SE2 does not currently have a driver for Safari (perhaps one is in the works...since there appears to be safari code in their repository?).  Should we just use SE1 since it supports all of the above browsers and then migrate to SE2 when it's more mature?

3.) We would like to use Grid to run tests in parallel but SE2 does not yet support it (though it appears that it may in the near future?).  I know you stated that HtmlDriver is faster, but we would be using the browser specific drivers so we wouldn't have this speed benefit.  Do you know when Grid will work for SE2?

Thanks for your thoughts!
December 19, 2010 8:00 PM
You need to sign in to comment on this blog
<August 2010>
SuMoTuWeThFrSa
25262728293031
1234567
891011121314
15161718192021
22232425262728
2930311234
Automated Script-generation with Powershell and SMO
 In the first of a series of articles on automating the process of building, modifying and copying SQL... Read more...

Converting String Data to XML and XML to String Data
 We all appreciate that, in general, XML documents or fragments are held in strings as text markup. In... Read more...

Geek of the Week: Don Syme
 With the arrival of F# 3.0 Microsoft announced a wide range of improvements such as type providers that... Read more...

How to Document and Configure SQL Server Instance Settings
 Occasionally, when you install identical databases on two different SQL Server instances, they will... Read more...

What's the Point of Using VARCHAR(n) Anymore?
 The arrival of the (MAX) data types in SQL Server 2005 were one of the most popular feature for the... Read more...