Table of Contents

• Exception Handling
• Conditional Statements (if/elif/else)
• For Loops
• Practical Web Scraping: Douban Movie Data
• Scraping All Top 250 Movies with Loop Control
• Handling Asynchronous Requests
• Working with Yellow Page 88 - Anti-Crawler Strategies

Exception Handling

Error Structure

When Python encounters an error, the runtime provides diagnostic information including:

• File name and line number
• Content of the problematic line
• Error type/classification

Handling Exceptions

Exception handling prevents program crashes when errors occur. The try...except block catches specific error types and allows graceful error management.

# Division by zero example
result = 10 / 0  # Raises ZeroDivisionError

# Input handling with type conversion
user_value = input('Enter a number: ')  # Returns string type
numeric_value = int(user_value) + 10    # TypeError if non-numeric input

Basic exception handling structure:

try:
    calculation = 1 / 0
except ZeroDivisionError:
    pass  # Handle gracefully, continue execution

When code between try and except executes without errors, normal flow continues. If an error occurs, the matching except block handles it.

# Multiple exception handling
try:
    sample_dict = {'key_a': 100}
    sample_dict['missing_key']
except ZeroDivisionError:
    print('Caught division error')
except KeyError:
    print('Caught key access error')

Catching multiple possible errors with generic exception:

try:
    user_input = input('Enter dictionary key: ')
    data = {'alpha': 1}
    value = data[user_input]
    division_result = 1 / 0
except Exception as error_message:
    print(f'Error occurred: {error_message}')
    print('Please check your input')

Note: Exception handling incurs performance overhead. Use only when necessary for robust error management.

Conditional Statements

The if statement handles decision-making scenarios in programming.

Single Branch

grade = 95
if grade > 90:
    print('Excellent')

Two Branches

grade = 85
if grade > 90:
    print('Excellent')
else:
    print('Good')

# Ternary expression equivalent
print('Excellent') if grade > 90 else print('Good')

Multiple Branches

Using elif (sequential evaluation):

score = 75
if score > 90:
    print('Grade A')
elif score > 60:
    print('Grade B')
else:
    print('Grade C')

Using nested if statements (independent evaluation):

score = 75
if score > 90:
    print('Grade A')
if score > 60 and score <= 90:
    print('Grade B')
if score <= 60:
    print('Grade C')

Key Difference

With elif structure, evaluation stops at the first matching condition. With nested if statements, each condition gets evaluated independently.

Example: When score = 91, the elif structure evaluates once and terminates, while nested if statements evaluate all three conditions, resulting in higher time and space complexity.

Logical Operators

>=    # Greater than or equal
<=    # Less than or equal
==     # Equal comparison
!=     # Not equal
not    # Logical negation
and    # Logical AND
or     # Logical OR

For Loops

For loops execute repetitive tasks efficiently. The loop iterates over sequences or ranges.

# Basic iteration over range
for counter in range(5):
    print(f'Iteration {counter}')

# Iterating over data collection
items = ['apple', 'banana', 'cherry']
for fruit in items:
    print(fruit)

Debugging tip: Set breakpoints and use debug mode to inspect variable values at each iteration.

Web Scraping Fundamentals

Core Scraping Workflow

1. Send Request: HTTP request to target URL
2. Retrieve Data: Receive server response
3. Parse Data: Extract required information
4. Store Data: Save to file or database

Key Libraries

Requests Library:

import requests

response = requests.get(url='https://example.com')
response.content    # Binary data
response.text       # Text content

Regular Expressions (re module):

import re

matches = re.findall('pattern', data_string, flags)
# Non-greedy pattern: (.*?)
# Greedy pattern: .*

File Operations:

with open('output.txt', 'w', encoding='utf-8') as file:
    file.write(content)

Scraping Douban Top 250 Movies

Scraping Essentials

The key to successful scraping involves understanding:

1. Website communication flow
2. Data source identification
3. Anti-crawling mechanisms
4. Countermeasures for protection

Pagination Analysis

Douban Top 250 uses pagination with URL pattern:

# Generate URLs for all pages
for page in range(10):
    start_index = page * 25
    url = f'https://movie.douban.com/top250?start={start_index}&filter='
    # Scraping logic continues here

Each page contains 25 movies, requiring 10 iterations to retrieve all 250 entries. Place the entire scraping logic inside the loop to process all pages sequentially.

Synchronous vs Asynchronous Requests

Synchronous Requests

The client sends a request and blocks execution until receiving the response. The thread cannot perform other tasks during this waiting period. This is similar to waiting for someone to confirm they heard you before continuing a conversation.

Asynchronous Requests

The client sends a request without waiting for response, allowing continued execution. Multiple operations proceed simultaneously. This is like sending a message and continuing work until you receive a notification.

Impact: Synchronous requests freeze the browser during waiting, while asynchronous requests maintain responsiveness.

Analyzing Asynchronous Data

Use browser developer tools:

1. Press F12 to open DevTools
2. Navigate to Network tab
3. Refresh the page
4. Filter by XHR to view asynchronous requests

Asynchronous data typically returns as JSON format - smaller data chunks without HTML markup.

JSON Data Handling

import json

# Convert Python object to JSON string
json_string = json.dumps(python_data)

# Convert JSON string to Python object
python_object = json.loads(json_string)

Handling Anti-Crawler Mechanisms

User-Agent Detection

Many websites identify bots by checking the User-Agent header. This identifies the client application making the request.

import requests

# Define browser-like headers
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}

# Use headers in request
response = requests.get(url='https://target-site.com', headers=headers)

Rotating User-Agent strings and adding other headers helps bypass basic anti-scraping measures.