Selenium-Based Approach

Direct HTTP requests to Baidu often trigger security verification challenges. Selenium provides browser automation capabilities for web testing and data extraction tasks. It supports multiple programming languages including Python, Java, and C# for controlling browser behavior through code that simulates user interactions like clicking links, filling forms, and submitting data.

Primary limitation: Slower execution speed compared to direct HTTP requests.

Environment Setup

1. Install Selenium library

pip install selenium -i https://pypi.tuna.tsinghua.edu.cn/simple

2. Install BeautifulSoup for parsing HTML content

pip install beautifulsoup4 -i https://pypi.tuna.tsinghua.edu.cn/simple

Browser Driver Installation

Download the appropriate browser driver for Selenium integration. For Firefox, obtain the latest geckodriver from Mozilla's GitHub repository (Windows users should select the win32 version).

Implementation Code

Import Required Modules

from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup

Driver Configuration

browser_options = Options()
browser_options.add_argument('-headless')  # Run browser in background
browser_options.binary_location = r'C:\Program Files\Mozilla Firefox\firefox.exe'  # Update with actual Firefox path
browser_options.set_capability('acceptInsecureCerts', True)

driver_service = Service(executable_path=r'./geckodriver.exe')  # Path to Firefox driver
browser_driver = webdriver.Firefox(service=driver_service, options=browser_options)

Content Exrtaction Strategy

Identify target content regions using browser developer tools. Press F12 to open developer tools, inspect elements to determine the specific area containing search results, and copy the CSS path of the containing element.

Baidu Search URL Format

search_term = "current date"  # Search query
search_url = f"https://www.baidu.com/s?wd={search_term}"

Text Extraction Process

Retrieve page source code:

browser_driver.get(search_url)
page_html = browser_driver.page_source
parsed_content = BeautifulSoup(page_html, 'html.parser')

Extract specific region content:

extracted_text = ''
content_selector = 'div#wrapper.wrapper_l.wrapper_new div#wrapper_wrapper div#container.sam_newgrid.container_l div#content_left'
matched_elements = parsed_content.select(content_selector)
for element in matched_elements:
    extracted_text += element.get_text(strip=True)
print("Extracted content:", extracted_text)

Complete Implementation

from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup

browser_options = Options()
browser_options.add_argument('-headless')
browser_options.binary_location = r'C:\Program Files\Mozilla Firefox\firefox.exe'
browser_options.set_capability('acceptInsecureCerts', True)

driver_service = Service(executable_path=r'./geckodriver.exe')
browser_driver = webdriver.Firefox(service=driver_service, options=browser_options)

search_query = "major news today"
target_url = f"https://www.baidu.com/s?wd={search_query}"

browser_driver.get(target_url)
page_source = browser_driver.page_source
html_parser = BeautifulSoup(page_source, 'html.parser')

result_text = ''
css_selector = 'div#wrapper.wrapper_l.wrapper_new div#wrapper_wrapper div#container.sam_newgrid.container_l div#content_left'
content_elements = html_parser.select(css_selector)
for item in content_elements:
    result_text += item.get_text(strip=True)
print("Search results:", result_text)

browser_driver.quit()

Requests-Based Approach

Request Headers

HTTP request headers transmit metadata between clients and servers, including environment details, preferences, and authentication data. Without proper headers, Baidu requests may fail or trigger security verification, particularly when cookies are absent.

Obtainn complete header information by performing a search in the browser, opening developer tools (F12), selecting the network request, and copying the request headers.

Python header format:

request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Cookie': 'your_cookie_here',
    'Accept': 'image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Host': 'www.baidu.com'
}

Basic implementation with headers:

import requests

search_phrase = "date"
baidu_url = f"https://www.baidu.com/s?wd={search_phrase}"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Cookie': 'your_cookie_here',
    'Accept': 'image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Host': 'www.baidu.com'
}
server_response = requests.get(baidu_url, headers=headers)
html_data = server_response.text
print(html_data)

HTML Parsing

Target Element Identification

Baidu search results appear within div elements with classes 'result' or 'result-op', both containing the 'c-container' subclass. Advertisement results lack specific class attributes but share the same structural level.

Example CSS path:

html body.cos-pc div#wrapper.wrapper_l.wrapper_new div#wrapper_wrapper div#container.sam_newgrid.container_l div#content_left div#1.result-op.c-container.xpath-log.new-pmd

Content Extraction

Use class selectors with BeautifulSoup to target relevant elements without specific IDs:

element_selector = 'div.result.c-container'

soup_parser = BeautifulSoup(html_data, 'html.parser')
selected_elements = soup_parser.select(element_selector)
output_content = ''
if selected_elements:
    for element in selected_elements:
        output_content += element.get_text(strip=True) + '\n'

print(output_content)

Complete Implementation

import requests
from bs4 import BeautifulSoup

query_text = "today's date"
search_url = f"https://www.baidu.com/s?wd={query_text}"

header_config = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Cookie': 'your_cookie_here',
    'Accept': 'image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Host': 'www.baidu.com'
}

response_data = requests.get(search_url, headers=header_config)
html_content = response_data.text

soup_instance = BeautifulSoup(html_content, 'html.parser')
selector_pattern = 'div.result.c-container'
result_elements = soup_instance.select(selector_pattern)
final_output = ''
if result_elements:
    for result_item in result_elements:
        final_output += result_item.get_text(strip=True) + '\n'

print(final_output)