How to Capture HTTP Messages with WebDriver

To capture HTTP Messages and network traffic with Selenium WebDriver we need to use an HTTP proxy, and configure the browser to direct all traffic through the proxy.

For this example I’m using an in code library as the HTTP proxy because that makes control of the test eaasier.

But this will work with any external proxy, you just have to start the proxy before you start the test.

HTTP Proxy

I’m using BrowserMob Proxy.

Which I add to my pom.xml as a dependency:

    <dependency>
        <groupId>net.lightbody.bmp</groupId>
        <artifactId>browsermob-core</artifactId>
        <version>2.1.5</version>
        <scope>test</scope>
    </dependency>

Page Using XHTTP Requests

For this example I am using:

https://testpages.herokuapp.com/styled/sync/xhttp-messages.html

Which, if you look in the Network traffic tab of your Browser Dev Tools, makes some XHTTP Requests (XHR) to update the page.

With a proxy we should be able to capture that traffic and extend the reach of our assertions.

BrowserMobProxy configuration

Start the proxy:

    proxyServer = new BrowserMobProxyServer();
    proxyServer.start();

    // capture content as a HAR (HTTP Archive)
    // to process when test is complete
    proxyServer.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT,
                                      CaptureType.RESPONSE_CONTENT);
    proxyServer.newHar();

And create the proxy config:

    final Proxy proxyConfig = ClientUtil.createSeleniumProxy(proxyServer);

General Proxy configuration

To configure a proxy in general we configure the proxy details:

    String proxyDetails = "127.0.0.1:8080";
    final Proxy proxyConfig = new Proxy().
                                setHttpProxy(proxyDetails).
                                setSslProxy(proxyDetails);

Configuring the Driver for the Proxy

And use those to configure the driver:

    final FirefoxOptions firefoxOptions = new FirefoxOptions();
    firefoxOptions.setProxy(proxyConfig);
    driver = new FirefoxDriver(firefoxOptions);

For Chrome I set the additional config for insecure certs:

    final ChromeOptions options = new ChromeOptions();
    options.setProxy(proxyConfig);
    options.setAcceptInsecureCerts(true);
    driver = new ChromeDriver(options);

Then do your ’test’ stuff

The test should run as normal, but now all traffic is going through the proxy.

Checking HTTP Traffic

For an external proxy you will then review the traffic externally in the proxy.

For BrowserMobProxy, because it is running in memory we can process the HTTP messages as part of the test.

    final Har httpMessages = proxyServer.getHar();
    for(HarEntry httpMessage : httpMessages.getLog().getEntries()){

        // check no errors on the XHR requests
        if(httpMessage.getRequest().getUrl().contains("/messageset")) {
            Assertions.assertEquals(200, httpMessage.getResponse().getStatus());
        }
    }

BrowserMobProxy has a lot more configuration and usage approaches, but the above is the simplest use of gathering traffic as you test.