Introduction

Recently, I was assigned a task to create a web automation script at work. This was my first attempt at writing such a script. While some basic operations were manageable using Selenium alone, automating slider captcha verification proved to be challenging. After combining Selenium with OpenCV, I was able to solve this problem successfully. Although there are many solutions for slider captcha automation online, few use Java, making this worth documenting.

1. Project Environment Setup

1.1 Installing and Integrating OpenCV Library

Refer to my other article for detailed instructions on setting up OpenCV in Java environment.

1.2 Adding Selenium Dependency

Using Maven to include Selenium, add the following dependency to your pom.xml:

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.21.0</version>
</dependency>

1.3 Downloading Browser Driver

Web automation requires a browser driver. I use Google Chrome, so I need to download ChromeDriver. First, check your browser's version, then download the matching driver version.

After extraction, you will get chromedriver.exe, which is the required driver.

2. Implementation

2.1 OpenCV Core Implementation

The key to slider captcha automation is obtaining accurate position information. Once we have the position data, we can use Selenium to move the slider to the correct location. OpenCV is used here to process images by comparing the background image with the slider image contours and finding the matching coordinates.

Before writing OpenCV code, we need to load the OpenCV library:

// Load the OpenCV native library
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

After loading, we need to read the background and slider images. This can be done using Selenium, which will be covered later. Let's proceed with the OpenCV implementation.

Reading Background and Slider Images

// Background image path
String backgroundPath = "path/to/background.jpg";
// Slider image path
String sliderPath = "path/to/slider.jpg";

// Use Mat class to read background and slider images
Mat backgroundImage = Imgcodecs.imread(backgroundPath);
Mat sliderImage = Imgcodecs.imread(sliderPath);

Converting Images to Grayscale - Critical Step

// Create Mat objects for grayscale images
Mat bgGrayscale = new Mat();
Mat sliderGrayscale = new Mat();

// Convert background image to grayscale
Imgproc.cvtColor(backgroundImage, bgGrayscale, Imgproc.COLOR_BGR2GRAY);
// Convert slider image to grayscale
Imgproc.cvtColor(sliderImage, sliderGrayscale, Imgproc.COLOR_BGR2GRAY);

// Save grayscale images
Imgcodecs.imwrite("./bg_grayscale.png", bgGrayscale);
Imgcodecs.imwrite("./slider_grayscale.png", sliderGrayscale);

Converting Grayscale to Edge Detection - Critical Step

// Create Mat objects for edge-detected images
Mat bgEdges = new Mat();
Mat sliderEdges = new Mat();

// Apply Canny edge detection
// The threshold values may need adjustment for clearer contours
Imgproc.Canny(bgGrayscale, bgEdges, 20, 200);
Imgproc.Canny(sliderGrayscale, sliderEdges, 20, 200);

// Save edge-detected images
Imgcodecs.imwrite("./bg_edges.png", bgEdges);
Imgcodecs.imwrite("./slider_edges.png", sliderEdges);

The above two steps are crucial; skipping them will result in inaccurate matching. After running the code, the effects are as follows:

// Perform template matching and save result
Mat resultImage = new Mat();
Imgproc.matchTemplate(bgEdges, sliderEdges, resultImage, Imgproc.TM_CCOEFF_NORMED);

// Get matching results
Core.MinMaxLocResult minMaxResult = Core.minMaxLoc(resultImage);
double minimumValue = minMaxResult.minVal;
double maximumValue = minMaxResult.maxVal;
Point minPoint = minMaxResult.minLoc;
Point maxPoint = minMaxResult.maxLoc;

// Get template (slider) image width and height
int templateWidth = sliderEdges.cols();
int templateHeight = sliderEdges.rows();

// Top-left corner coordinates
Point topLeft = maxPoint;
// Calculate bottom-right corner
Point bottomRight = new Point(topLeft.x + templateWidth, topLeft.y + templateHeight);

// Draw rectangle on original background
Imgproc.rectangle(backgroundImage, topLeft, bottomRight, new Scalar(0, 0, 255), 2);
// Save result image
Imgcodecs.imwrite("./result.png", backgroundImage);

The result shows the gap position accurately. With some mathematical calculations, we can obtain the offset needed.

Now let's create a reusable method (note: the example reads images from HTTP URLs):

public double calculateSliderOffset(String backgroundUrl, String sliderUrl) {
    System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

    // Read images from URL
    Mat backgroundMat = loadImageFromNetwork(backgroundUrl);
    Mat sliderMat = loadImageFromNetwork(sliderUrl);

    // Verify images loaded successfully
    if (backgroundMat.empty() || sliderMat.empty()) {
        System.out.println("Error: Image cannot be loaded!");
        return 0;
    }

    // Convert to grayscale
    Mat bgGray = new Mat();
    Mat sliderGray = new Mat();
    Imgproc.cvtColor(backgroundMat, bgGray, Imgproc.COLOR_BGR2GRAY);
    Imgproc.cvtColor(sliderMat, sliderGray, Imgproc.COLOR_BGR2GRAY);

    // Edge detection
    Mat bgEdge = new Mat();
    Mat sliderEdge = new Mat();
    Imgproc.Canny(bgGray, bgEdge, 20, 200);
    Imgproc.Canny(sliderGray, sliderEdge, 20, 200);

    // Template matching
    Mat matchResult = new Mat();
    Imgproc.matchTemplate(bgEdge, sliderEdge, matchResult, Imgproc.TM_CCOEFF_NORMED);

    // Get matching results
    Core.MinMaxLocResult mlResult = Core.minMaxLoc(matchResult);
    Point matchPoint = mlResult.maxLoc;

    // Get template dimensions
    int tWidth = sliderEdge.cols();
    int tHeight = sliderEdge.rows();

    // Calculate coordinates
    Point startPoint = matchPoint;
    Point endPoint = new Point(startPoint.x + tWidth, startPoint.y + tHeight);

    // Draw rectangle for visualization
    Imgproc.rectangle(backgroundMat, startPoint, endPoint, new Scalar(0, 0, 255), 2);
    Imgcodecs.imwrite("./result.png", backgroundMat);

    // Return gap position (center point)
    return startPoint.x + tWidth / 2;
}

private Mat loadImageFromNetwork(String imageUrl) {
    try {
        URL url = new URL(imageUrl);
        byte[] imageData = downloadImage(url.openStream());
        ByteBuffer buffer = ByteBuffer.wrap(imageData);
        return Imgcodecs.imdecode(new Mat(1, imageData.length, Core.CV_8UC1, buffer), Imgcodecs.IMREAD_COLOR);
    } catch (IOException e) {
        e.printStackTrace();
        return null;
    }
}

private byte[] downloadImage(java.io.InputStream inputStream) throws IOException {
    byte[] buffer = new byte[inputStream.available()];
    int bytesRead;
    int total = 0;
    while ((bytesRead = inputStream.read(buffer, total, buffer.length - total)) > 0) {
        total += bytesRead;
        if (total == buffer.length) {
            buffer = Arrays.copyOf(buffer, buffer.length * 2);
        }
    }
    return Arrays.copyOf(buffer, total);
}

2.2 Moving Slider with Selenium

Now that we have the offset from OpenCV, we can use Selenium to simulate the slider movement:

// Load driver
String driverPath = "path/to/chromedriver.exe";
System.setProperty("webdriver.chrome.driver", driverPath);

// Open browser
WebDriver driver = new ChromeDriver();
// Navigate to URL
driver.get("http://example.com");

// Get background and slider image elements
WebElement bgImageElement = driver.findElement(By.xpath("//div[@id='background']"));
WebElement sliderImageElement = driver.findElement(By.xpath("//div[@id='slider']"));

// Get image URLs
String bgUrl = bgImageElement.getAttribute("src");
String sliderUrl = sliderImageElement.getAttribute("src");

// Get slider button element
WebElement sliderButton = driver.findElement(By.xpath("//div[@class='slider-handle']"));

// Calculate offset using the method created earlier
double offset = calculateSliderOffset(bgUrl, sliderUrl);

if (sliderButton != null) {
    Actions actionBuilder = new Actions(driver);
    
    // Divide movement into steps to avoid detection
    int stepCount = 20;
    int stepSize = (int) (offset / stepCount);
    
    // Click and start moving
    actionBuilder.click(sliderButton).perform();
    
    for (int i = 1; i <= stepCount; i++) {
        actionBuilder.moveByOffset(stepSize, 0).perform();
        Thread.sleep(20); // Control movement speed
    }
    actionBuilder.release().perform();
}

The implemantation is now complete. While this is a basic example, real-world usage would require additional algorithms to reduce offset errors for better success rates.