Overview

• Recursive crawling and parsing of multi-page data
• Scrapy core component workflow
• Sending POST requests in Scrapy

Recursive Multi-Page Data Crawling

Requirement: Crawl all page from Qiushibaiek and extract author names and joke content, then persist the data.

Analysis: Each page corresponds to a unique URL. The Scrapy project needs to request each page URL sequentially and parse the author and content using the appropriate parsing method.

Implementation Approaches:

1. Add each page URL to the start_urls list in the spider file. (Not recommended)
2. Manually create requests using the Request method. (Recommended)

Code Example:

# -*- coding: utf-8 -*-
import scrapy
from qiushibaike.items import QiushibaikeItem

class JokeSpider(scrapy.Spider):
    name = 'jokes'
    allowed_domains = ['www.qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/']

    # Track current page number
    current_page = 1
    page_template = 'https://www.qiushibaike.com/text/page/%s/'

    def parse(self, response):
        # Extract all joke blocks from the page
        joke_blocks = response.xpath('//*[@id="content-left"]/div')
        
        for block in joke_blocks:
            # Extract author name
            author = block.xpath('.//div[@class="author clearfix"]//h2/text()').extract_first()
            if author:
                author = author.strip('\n')
            
            # Extract joke content
            content = block.xpath('.//div[@class="content"]/span/text()').extract_first()
            if content:
                content = content.strip('\n')
            
            # Create item and populate with data
            item = QiushibaikeItem()
            item['author'] = author
            item['content'] = content

            # Yield item to pipeline for persistence
            yield item

        # Continue crawling remaining pages
        if self.current_page < 13:
            self.current_page += 1
            next_url = self.page_template % self.current_page

            # Recursive request: callback directs where to send parsed response
            yield scrapy.Request(url=next_url, callback=self.parse)

Scrapy Core Components Workflow

The Scrapy framework consists of five core components that work together:

1. Engine
Handles the entire system data flow and triggers transactions. This is the core of the framework.

2. Scheduler
Receives requests from the engine, pushes them into a queue, and returns them when the engine requests again. Functions as a priority queue for URLs, determining which URL to crawl next while removing duplicates.

3. Downloader
Dwonloads web page content and returns it to the spider. Built on Twisted, an efficient asynchronous model.

4. Spiders
The main working component. Extracts required information (Items) from specific web pages. Can also extract links for Scrapy to continue crawling subsequent pages.

5. Pipeline
Processes entities extracted by spiders. Primary functions include persisting entities, validating entity validity, and cleaning unnecessary data. When a page is parsed by the spider, it is sent to the pipeline where data undergoes several sequential processing stages.

Sending POST Requests

Question: In previous examples, we never manually sent requests for URLs in the start_urls list, yet requests were indeed made. How does this work?

Answer: The spider class inherits the start_requests(self) method from the Spider parent class. This method automatically initiates requests for all URLs in the start_urls list:

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(url=url, callback=self.parse)

Note: By default, this method sends GET requests for start URLs. To send POST requests, you need to override this method.

Method: Override start_requests to send POST requests:

def start_requests(self):
    # Target URL for POST request
    endpoint = 'http://fanyi.baidu.com/sug'
    
    # POST request payload
    payload = {
        'kw': 'wolf',
    }
    
    # Send POST request using FormRequest
    yield scrapy.FormRequest(url=endpoint, formdata=payload, callback=self.parse)

The FormRequest class is specifically designed for handling form submissions and POST requests in Scrapy.