Capturing Network Logs (HAR Files) with Selenium

Introduction

Ever been in a situation where you needed to get down to the nitty-gritty of network activity on your web application? Well, you're in the right place. Today, I’m going to walk you through how to capture network logs in the HAR (HTTP Archive) format using Selenium and the Chrome DevTools Protocol (CDP) in Java. This will be a game-changer for debugging and performance analysis.

Prerequisites

Before we dive in, make sure you’ve got these essentials:

Java Development Kit (JDK) - the latest version is preferred.
Selenium library (selenium-java).
ChromeDriver - ensure it matches your version of Chrome.
A build tool like Maven or Gradle to manage dependencies.

Step 1: Add Selenium Dependency

If you’re using Maven, add this to your pom.xml:

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.1.0</version>
</dependency>

For Gradle users, add this to your build.gradle:

dependencies {
    testImplementation 'org.seleniumhq.selenium:selenium-java:4.1.0'
}

Step 2: Set Up ChromeDriver

Time to set up ChromeDriver and make it work with CDP. Here’s a simple code snippet to get you started:

import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v85.network.Network;
import org.openqa.selenium.devtools.v85.network.model.RequestWillBeSent;
import org.openqa.selenium.devtools.v85.network.model.ResponseReceived;
import org.openqa.selenium.devtools.v85.network.model.Response;

public class CaptureHAR {
    public static void main(String[] args) {
        // Set the path to the ChromeDriver executable
        System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");

        // Set Chrome options
        ChromeOptions options = new ChromeOptions();

        // Initialize ChromeDriver
        ChromeDriver driver = new ChromeDriver(options);

        // Create DevTools session
        DevTools devTools = driver.getDevTools();
        devTools.createSession();

        // Enable Network
        devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));

        // Add event listener to capture network requests
        devTools.addListener(Network.requestWillBeSent(), request -> {
            System.out.println("Request: " + request.getRequest().getUrl());
        });

        // Add event listener to capture network responses
        devTools.addListener(Network.responseReceived(), response -> {
            Response res = response.getResponse();
            System.out.println("Response: " + res.getUrl() + " - " + res.getStatus());
        });

        // Navigate to the website
        driver.get("https://example.com");

        // Perform any actions on the website here

        // Close the browser
        driver.quit();
    }
}

Step 3: Capture and Save HAR Logs

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import org.openqa.selenium.devtools.v85.network.model.ResponseReceived;

import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

public class CaptureHAR {
    public static void main(String[] args) {
        System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");

        ChromeOptions options = new ChromeOptions();
        ChromeDriver driver = new ChromeDriver(options);

        DevTools devTools = driver.getDevTools();
        devTools.createSession();

        devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));

        Gson gson = new GsonBuilder().setPrettyPrinting().create();

        List<ResponseReceived> responses = new ArrayList<>();

        devTools.addListener(Network.responseReceived(), responses::add);

        driver.get("https://example.com");

        // Wait for some time to capture network activity
        try {
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        try (FileWriter writer = new FileWriter("network_logs.har")) {
            gson.toJson(responses, writer);
        } catch (IOException e) {
            e.printStackTrace();
        }

        driver.quit();
    }
}

Wrapping Up

And there you have it! Capturing network logs using Selenium and Chrome DevTools Protocol in Java is a powerful tool for web developers. Whether you’re debugging an issue or optimizing performance, these HAR files will provide invaluable insights into your web application's network activity.