Handling Dynamic Image Loading

Standard HTTP requests often fail to retrieve assets on modern websites due to optimization strategies like lazy loading. This technique delays image requests until the user scrolls into view, reducing initial bandwidth consumption. Consequently, the src attribute in the HTML source may remain empty or point to a placeholder, with the actual URL stored in custom data attributes (e.g., src2, data-original).

To extract these resources programmatically, one must inspect the DOM after rendering or identify the fallback attribute keys.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
from lxml import etree

def scrape_gallery_images():
    target_link = 'http://sc.chinaz.com/tupian/gudianmeinvtupian.html'
    http_headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    }
    
    resp = requests.get(url=target_link, headers=http_headers)
    resp.encoding = 'utf-8'
    raw_html = resp.text

    tree_structure = etree.HTML(raw_html)
    content_blocks = tree_structure.xpath('//div[@id="container"]/div')

    for block in content_blocks:
        # Retrieving the specific pseudo-attribute used for lazy loading
        src_addresses = block.xpath('.//img/@src2')
        alt_names = block.xpath('.//img/@alt')
        
        print(src_addresses)
        print(alt_names)

if __name__ == '__main__':
    scrape_gallery_images()

By examining the network response during development, developers can pinpoint the correct attribute names required for successful extraction without executing JavaScript.

Browser Automation Fundamentals

When static analysis fails, automation tools like Selenium provide the ability to control browser instances directly. This enables interaction with dynamic elements that require JavaScript execution, such as clicking pagination buttons or filling forms before data loads.

Configuration Prerequisites

1. Library Installation: Execute pip install selenium.
2. Driver Management: Download a WebDriver binary compatible with your installed browser version (e.g., Chromedriver).
3. Path Verification: Ensure the driver executable path is correctly configured within the script environment.

Interaction Logic

The following snippet demonstrates navigtaing a search interface using programmatic selectors.

from selenium import webdriver
import time

# Initialize controller pointing to driver binary
chrome_driver_path = r'C:\Automation\chromedriver.exe'
browser_control = webdriver.Chrome(executable_path=chrome_driver_path)

try:
    # Navigate to base URL
    browser_control.get("http://www.baidu.com")
    
    # Locate menu items dynamically
    settings_item = browser_control.find_element_by_link_text('设置')
    if settings_item:
        settings_item.click()
        time.sleep(2)

    # Adjust preferences via navigation
    search_pref = browser_control.find_element_by_link_text('搜索设置')
    if search_pref:
        search_pref.click()
        time.sleep(2)

    # Modify display configuration
    count_selector = browser_control.find_element_by_id('nr')
    if count_selector:
        # Select third option (50 results per page)
        option_node = count_selector.find_element_by_xpath('//*[@id="nr"]/option[3]')
        option_node.click()
        time.sleep(2)

    # Submit changes
    submit_ctrl = browser_control.find_element_by_class_name("prefpanelgo")[0]
    submit_ctrl.click()

    time.sleep(2)

    # Handle modal alerts gracefully
    alert_handler = browser_control.switch_to.alert
    alert_handler.accept()

    # Input query string
    input_box = browser_control.find_element_by_id('kw')
    input_box.send_keys('QueryKeyword')
    time.sleep(1)

    # Trigger submission
    search_trigger = browser_control.find_element_by_id('su')
    search_trigger.click()

    time.sleep(3)

except Exception as e:
    print(f"Process interrupted: {e}")
finally:
    browser_control.quit()

Available element locators include find_element_by_id, find_elements_by_class_name, find_element_by_xpath, and find_element_by_name.

Headless Execution with PhantomJS

Running visible browsers can consume significant resources. PhantomJS was historically used as a headless engine to simulate browser behavior in the background. It supports screenshot capture functions to verify state transitions during automated runs.

from selenium import webdriver
import time

phantom_exe = r'C:\Engine\phantomjs.exe'
hl_browser = webdriver.PhantomJS(executable_path=phantom_exe)

page_url = 'http://www.baidu.com/'
hl_browser.get(page_url)

time.sleep(3)

# Snapshot current state
hl_browser.save_screenshot(r'snapshot_before.png')

cmd_input = hl_browser.find_element_by_id('kw')
cmd_input.send_keys('TargetTerm')

time.sleep(3)

# Verify input visually
hl_browser.save_screenshot(r'snapshot_after.png')

search_btn = hl_browser.find_element_by_class_name('s_btn')[0]
search_btn.click()

time.sleep(3)
hl_browser.quit()

Scrolling for Infinite Content

Web applications often load more entries via scroll events rather than distinct pages. Executing JavaScript commands allows the scraper to simulate scrollling to the bottom of the page, triggering the additional data fetch.

from selenium import webdriver
import time

dynamic_url = 'https://movie.douban.com/typerank?type_name=%E6%81%90%E6%80%96&type=20&interval_id=100:90&action='
exec_path = r'C:\Engine\phantomjs.exe'

auto_bot = webdriver.PhantomJS(executable_path=exec_path)
auto_bot.get(dynamic_url)

time.sleep(3)
auto_bot.save_screenshot('view_1.png')

# Scroll logic simulation
scroll_js = "window.scrollTo(0, document.body.scrollHeight)"
auto_bot.execute_script(scroll_js)
time.sleep(2)

auto_bot.execute_script(scroll_js)
time.sleep(2)
auto_bot.save_screenshot('view_2.png')

# Extract rendered HTML
dump_content = auto_bot.page_source
with open('./fetched_data.html', 'w', encoding='utf-8') as writer:
    writer.write(dump_content)

auto_bot.quit()

Modern Headless Alternatives: Chrome Options

Following maintenance discontinuation for PhantomJS, Google Chrome’s native headless mode has become the standard recommendation. It utilizes the same rendering engine as the standard browser but operates without the UI component.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

# Setup configuration object
chrome_cfg = Options()
chrome_cfg.add_argument('--headless')
chrome_cfg.add_argument('--disable-gpu')

driver_bin = r'C:\Automation\chromedriver.exe'

browser_headless = webdriver.Chrome(executable_path=driver_bin, options=chrome_cfg)

url_target = 'http://www.baidu.com/'
browser_headless.get(url_target)

time.sleep(3)
browser_headless.save_screenshot('capture_final.png')

browser_headless.quit()

This method ensures access to up-to-date web standards while maintaining the efficiency required for server-side crawling operations.

Practice Task

Apply the techniques learned above to extract headlines and article bodies from domestic news sections on major media portals. Pay attention to any authentication requirementss or anti-scraping mechanisms that may be triggered during the process.