Running Selenium tests with multiple WebRunners

By YopY on Monday 16 May 2011 11:49 - Comments (3)
Category: -, Views: 7.314

Note: This is a crosspost from Xebia neXt. Read the original here. Crossposting here for shameless self-promotional purposes.

We decided to use Selenium to run front-end integration tests for our current project, and I was charged to investigate, seeing that nobody in the team had ever worked with Selenium before. I did, for my previous employer, but it was limited to playing around with Selenium IDE for a day - we never found the value in developing a full integration test for the dozen-or-so corporate websites we managed, in part because we didn't really know /what/ to test.

In any case, I did some browsing and, as it turns out, Selenium 2.0 is under heavy development, currently in beta stage. Selenium 2.0 is the combined effort to merge the original Selenium project with the WebDriver project, a means of running Selenium tests from (Java) unit tests, which 'remote controls' a browser running on the local system. Here's a simple example of what a Selenium test with WebDriver looks like:


Java:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class WebDriverSupplier {

  private static final Supplier<WebDriver> FIREFOX_SUPPLIER = new Supplier<WebDriver>() {
    @Override
    public WebDriver get() {
      return new FirefoxDriver();
    }
  };

  private static final Supplier<WebDriver> INTERNET_EXPLORER_SUPPLIER = new Supplier<WebDriver>() {
    private WebDriver ieDriver;
    @Override
    public WebDriver get() {
      return new InternetExplorerDriver();
    }
  };

  private static final List<Supplier<WebDriver>> suppliers = new ArrayList<Supplier<WebDriver>>();

  static {
    if (System.getProperty("os.name").startsWith("Windows")) {
      suppliers.add(INTERNET_EXPLORER_SUPPLIER);
    }
    suppliers.add(FIREFOX_SUPPLIER);
  }

  public static List<Supplier<WebDriver>> getWebDrivers() {
    return suppliers;
  }
}


gist

It looks quite straightforward - you instruct WebDriver to open up a browser window by creating a new implementation of WebDriver, send 'commands' to it, evaluate responses, andsoforth. When running the test, you will see a browser window open and various actions being performed, based on the input from the unit test. Finally, by calling WebDriver.close(), the browser window is closed.

Now. A good integration test will run the tests on each officially supported platform. In our case, this would be Internet Explorer 7 and Firefox version 3.x. Ideally, we would run the integration tests against these exact platforms, however, our team turned out to be too modern - I was the only Windows user, and already was at Internet Explorer 9. The others had all upgraded to Firefox 4. Compatibility problems ensued. Initially, I had added Selenium 2.0a4, but as it turns out, the WebDriver(s) in that version didn't support either IE9 or Firefox 4.

We were at a standstill for a moment then, but after some creative Googling and a 'duh' moment, it turns out we were horribly outdated. A few versions later, version 2.0b2, Internet Explorer 9 and Firefox 4 support was added.

On to the next problem. I considered it a good thing to run the integration tests on all supported platforms, so I first devised a structure that retrieved a list of WebDrivers to run all the tests with. This became a factory-like class as you see below. It returns a List of Supplier objects, one of Google's utility interfaces that does little else than return an instance of an object, making it suitable for use as a factory, generator, builder, closure, or something else entirely. Each Supplier returns a new instance of its matching WebDriver when called upon.


Java:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class WebDriverSupplier {

  private static final Supplier<WebDriver> FIREFOX_SUPPLIER = new Supplier<WebDriver>() {
    @Override
    public WebDriver get() {
      return new FirefoxDriver();
    }
  };

  private static final Supplier<WebDriver> INTERNET_EXPLORER_SUPPLIER = new Supplier<WebDriver>() {
    private WebDriver ieDriver;
    @Override
    public WebDriver get() {
      return new InternetExplorerDriver();
    }
  };

  private static final List<Supplier<WebDriver>> suppliers = new ArrayList<Supplier<WebDriver>>();

  static {
    if (System.getProperty("os.name").startsWith("Windows")) {
      suppliers.add(INTERNET_EXPLORER_SUPPLIER);
    }
    suppliers.add(FIREFOX_SUPPLIER);
  }

  public static List<Supplier<WebDriver>> getWebDrivers() {
    return suppliers;
  }
}


gist

Using all the registered WebDrivers in an integration test involved running the body of the tests in a loop, for each WebDriver, run these tests. I'm still wondering if there's a neater solution for this, an annotation maybe that indicates to 'run this test using all registered WebDrivers. But this will do.

View example on Gist

Of course, we don't want errors to occur when people without Internet Explorer try to run it, so we added a line that makes sure the IE WebDriver is only added if the user is running Windows:


Java:
1
2
3
if (System.getProperty("os.name").startsWith("Windows"))
  suppliers.add(INTERNET_EXPLORER_SUPPLIER)
}



This worked. For each integration test, a new, fresh browser window was opened, the tests were run, and the window was closed again.

Of course, it wasn't done yet. First, creating a new browser window and session for every test takes a (relative) large amount of time, making the integration tests run for at least twice as long as they should. Second, the Java binary crashed when it tried to start up an Internet Explorer window for the second time. It's known that you can't open two IE windows at the same time with the Selenium WebDriver, but it crashing when opening two windows in the same JVM session is, as far as I know, not a well-known problem.

Of course, we could solve two problems in one stroke by simply re-using the same browser windows for every test. Because we already had a factory structure in place, this was a matter of storing a single WebDriver instance in the Supplier class, initialize it on first call, and return it on consequent calls. Note that the following solution isn't threadsafe.


Java:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
public class WebDriverSupplier {

  private static final Supplier<WebDriver> FIREFOX_SUPPLIER = new Supplier<WebDriver>() {
    private WebDriver firefoxDriver;

    @Override
    public WebDriver get() {
      if (firefoxDriver == null) {
        firefoxDriver = new FirefoxDriver();
      }
      return firefoxDriver;
    }
  };

  private static final Supplier<WebDriver> INTERNET_EXPLORER_SUPPLIER = new Supplier<WebDriver>() {
    private WebDriver ieDriver;

    @Override
    public WebDriver get() {
      if (ieDriver == null) {
        ieDriver = new InternetExplorerDriver();
      }
      return ieDriver;
    }
  };

  private static final List<Supplier<WebDriver>> suppliers = new ArrayList<Supplier<WebDriver>>();

  static {
    if (System.getProperty("os.name").startsWith("Windows")) {
      suppliers.add(INTERNET_EXPLORER_SUPPLIER);
    }
    suppliers.add(FIREFOX_SUPPLIER);
  }

  public static List<Supplier<WebDriver>> getWebDrivers() {
    return suppliers;
  }
}


gist

Considering this is the solution we've been using for a couple of weeks now, I'd say it's done. I hope other people will find this post useful - I myself wasn't able to find any examples for running Selenium tests with multiple browser types.

Some final notes on Internet Explorer and Selenium:

Clicking links and buttons through WebDriver in Internet Explorer turns out to be a tricky business; adding various hacks in the integration tests themselves make it workable, but it's far from ideal. See a StackOverflow question about this issue. We've added a click method of our own as a workaround to this issue - it can click on any clickable element using the WebDriver syntax (the By parameter). view on gist.

I wasn't able to find out how to instruct Selenium to activate IE 7 compatibility mode in Internet Explorer 9, a requirement for making sure the test works properly in IE 7. One possible solution (and at the moment, the only solution I can see) is to include a meta-tag or HTTP header telling IE to run the site in IE 7 compatibility mode. However, this is a step easily forgotten when going to production, and may cause problems in the future, when people try to run it in IE 13 that, with any luck, no longer provides backwards compatibility with its old and quirky renderers.