Jsoup is a robust Java library designed for working with real-world HTML. It provides a convenient API for fetching URLs and extracting data using DOM methods or CSS selectors. This guide demonstrates how to scrape product listings from JD.com, specifically addressing the anti-crawler mechanisms that redirect requests to a security verification page.

Handling JD Anti-Scraping Measures

Direct requests to JD search URLs often result in a redirect to a "Security" page. This occurs because the server treats the request as coming from an unverified client. To bypass this, specific cookies—particularly the thor cookie—must be sent with the request.

To obtain the necessary cookies:

1. Open the target URL in a browser and navigate to the search results.
2. Open Developer Tools (F12) and go to the Network tab.
3. Refresh the page and find the main search request.
4. Inspect the Request Headers and copy the value of the thor cookie.

Implementing the Scraper

The following Java code illustrates the complete process. It initializes a cookie map, connects to the target URL with the required authentication, parses the HTML document, and iterates through product list items to extract specific attributes.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

public class JdProductScraper {

    // Target search URL for demonstration purposes (e.g., searching for tissues)
    private static final String SEARCH_URI = "https://search.jd.com/Search?keyword=%E9%A4%90%E5%B7%BE%E7%BA%B8";

    public static void main(String[] args) {
        try {
            // Prepare authentication headers to bypass the security check
            Map<String, String> authCookies = new HashMap<>();
            // Replace the value below with a valid 'thor' cookie retrieved from your browser
            authCookies.put("thor", "YOUR_COPIED_THOR_COOKIE_VALUE_HERE");

            // Establish connection and fetch the document
            Document doc = Jsoup.connect(SEARCH_URI)
                    .cookies(authCookies)
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
                    .timeout(10000)
                    .get();

            // Locate the container holding the product list
            Element productListContainer = doc.selectFirst("ul.gl-warp.clearfix");
            
            if (productListContainer != null) {
                Elements items = productListContainer.select("li.gl-item");

                for (Element item : items) {
                    extractProductDetails(item);
                }
            } else {
                System.out.println("Product list container not found. Check cookies or selectors.");
            }

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private static void extractProductDetails(Element item) {
        try {
            // Extract product image URL (lazy-loaded attribute)
            String imgUrl = item.selectFirst("img").attr("data-lazy-img");
            
            // Extract price text
            String price = item.selectFirst(".p-price").text();
            
            // Extract shop name
            String shopName = item.selectFirst(".p-shop").text();

            // Output the parsed data
            System.out.println("------------------------------");
            System.out.println("Image: " + imgUrl);
            System.out.println("Price: " + price);
            System.out.println("Shop: " + shopName);

        } catch (NullPointerException e) {
            // Handle cases where specific elements might be missing in the DOM structure
            System.out.println("Skipping item due to missing data structure.");
        }
    }
}

Data Extraction Logic

The code navigates the DOM structure by targeting the ul element with the class gl-warp clearfix. Within this container, individual products are represented by li tags. The extractProductDetails method queries these elements for specific classes:

• Image: The img tag's data-lazy-img attribute stores the actual image source used for lazy loading.
• Price: Located within a div or element containing the class p-price.
• Shop: Found inside the element with the class p-shop.