Understanding HTTP Verbs in Data Retrieval

Web interactions primarily rely on two HTTP methods: GET and POST. GET requests are stateless and idempotent, typically used for fetching resources directly via a URL. POST requests are designed for transmitting data to a server, often triggered by form submissions or API calls where parameters are embedded in the request body rather than the URI.

GET Request Implementation

import requests

# Standard resource retrieval
target_endpoint = "https://api.example.com/resources"
response = requests.get(target_endpoint)
decoded_content = response.content.decode("utf-8")

Text encoding varies by source. While utf-8 is the modern standard, legacy systems frequently use gbk, gb2312, or gb18030. Always inspect the response headers to determine the correct charset.

POST Request Implementation

When submitting payloads, the structure of the data dictionary must align with the server's expectations. For standard form-encoded submissions:

form_payload = {
    "search_term": "python_scraping",
    "page_index": 2
}
post_response = requests.post(target_endpoint, data=form_payload)
result_html = post_response.content.decode("gbk")

Modern APIs often require JSON payloads. In this scenario, replace the data parameter with json=form_payload to automatically set the Content-Type: application/json header.

Concurrency Models and Thread Management

The concurrent.futures.ThreadPoolExecutor (or the legacy multiprocessing.dummy module) provides a straightforward interface for parallelizing I/O-bound tasks. The map() function applies a target callable across an iterable of inputs, distributing work across a fixed pool of worker threads.

Performance scaling follows a non-linear curve. For small workloads, the overhead of thread creation and context switching negates any speedup compared to sequential execution. As task volume increases, threads initially improve throughput until CPU scheduling and the Global Interpreter Lock (GIL) introduce diminishing returns. Beyond this threshold, asynchronous event loops (asyncio) or multi-processing architectures become necessary to maintain efficiency.

Graph Traversal in Crawl Architecture

Web crawlers navigate hyperlink graphs using two fundamental algorithms:

• Depth-First Search (DFS): Explores a single branch to its maximum depth before backtracking. Implemented via a stack or recursion. Useful for deep directory structures but risks getting trapped in infinite loops if cycle detection is absent.
• Breadth-First Search (BFS): Explores all immediate neighbors before moving deeper. Implemented via a queue. Preferred for systematic site mapping and ensuring uniform discovery levels across a domain.

Practical Implementation: Structured Content Extraction

The following script demonstrates a complete pipeline for fetching a book catalog, resolving individual chapter URLs, parsing textual content, and persisting results to local storage. It utilizes regular expressions for pattern matching and modular function design for maintainability.

import requests
import re
import os

BASE_URI = "https://www.kanunu8.com/book3/6633/"
OUTPUT_DIRECTORY = "extracted_novels"

def retrieve_page(url, charset="gbk"):
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    return resp.content.decode(charset)

def extract_links(catalog_html):
    tbody_pattern = re.compile(r'<tbody>]*>(.*?)</tbody>', re.DOTALL)
    href_pattern = re.compile(r'href="([^"]+)"')
    
    match = tbody_pattern.search(catalog_html)
    if not match:
        return []
        
    return [f"{BASE_URI}{link}" for link in href_pattern.findall(match.group(1))]

def parse_chapter_data(page_html):
    title_regex = re.compile(r'size="4">(.*?)<', re.DOTALL)
    title_match = title_regex.search(page_html)
    chapter_title = title_match.group(1).strip() if title_match else "untitled"
    
    content_regex = re.compile(r'<p>(.*?)</p>', re.DOTALL)
    content_match = content_regex.search(page_html)
    if content_match:
        raw_text = content_match.group(1)
        cleaned_text = raw_text.replace("<br></br>", "\n")
        return chapter_title, cleaned_text
    return chapter_title, ""

def write_to_file(chapter, content):
    os.makedirs(OUTPUT_DIRECTORY, exist_ok=True)
    file_path = os.path.join(OUTPUT_DIRECTORY, f"{chapter}.txt")
    with open(file_path, "w", encoding="utf-8") as f:
        f.write(content)

def execute_pipeline():
    catalog = retrieve_page(BASE_URI)
    chapter_links = extract_links(catalog)
    print(f"Initiating extraction for {len(chapter_links)} chapters...")
    
    for link in chapter_links:
        try:
            page_data = retrieve_page(link)
            name, body = parse_chapter_data(page_data)
            write_to_file(name, body)
        except Exception as err:
            print(f"Skipped {link} due to error: {err}")
    print("Extraction pipeline completed.")

if __name__ == "__main__":
    execute_pipeline()

Common Development Pitfalls

• Environment Isolation: Integrated development environments often default to system-wide Python installations. Ensure your IDE interpreter points to the active virtual environment where third-party packages are installed.
• HTTP Method Misapplication: Blindly switching between GET and POST without inspecting network traffic leads to 403/405 errors. Use browser developer tools to verify the exact request type and payload format.
• Data Type Confusion: Parsing functions expect raw HTML strings, not URL strings. Passing endpoints directly to regex extractors will yield empty results.
• Exception Handling Neglect: Uncaught requests.exceptions.RequestException or IndexError from failed regex matches will crash the script. Always implement try-except blocks and validate regex matches before accessing group indices.