Throttle Request Frequency

Many websites flag crawlers that send requests at super-human speeds, such as non-stop page crawling, rapid form submissions, or bulk image downloading. Two common approaches to add natural browsing delays are fixed waiting and conditinoal waiting.

Fixed Delay

This forces the crawler to pause for a set interval between requests:

import time
# Pause for 3 seconds between consecutive requests
time.sleep(3)

For large crawls, consider running jobs during off-peak hours and keep your request rate low to avoid being detected as a bot.

Conditional Implicit Wait

This approach waits until all required page elements finish loading before proceeding, which mimics real human browsing behavior and avoids errors from incomplete page loads. A common pattern with Selenium uses wait.until():

page_wait.until(lambda driver: driver.find_element_by_xpath("//div[@id='link-report']/span"))

This line only continues execution once the target element loads fully, preventing overly fast crawling that triggers anti-bot systems.

Spoof Request Headers

The User-Agent request header is one of the most common checks sites use to distinguish real browsers from automated crawlers. Default User-Agent strings for Python HTTP libraries clearly mark requests as automated, so you should override this header with a valid browser User-Agent:

import urllib2

target_url = "https://example.com"
request = urllib2.Request(target_url)
# Add a valid browser User-Agent to spoof a real browsing session
request.add_header(
    'User-Agent',
    'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36'
)

response = urllib2.urlopen(request)

Use Rotating Proxy IP Addresses

If your IP address gets blocked by a site, rotating proxies is the most effective workaround. It is best practice to use proxies from the start of large crawls to avoid IP blocks entirely.

Below is a basic example of configuring a single proxy with urllib2, tested on a site that displays your current public IP:

# -*- coding: utf-8 -*-
import urllib2

test_url = "http://www.ip181.com/"
# Configure proxy: dictionary format {protocol: proxy_ip:port}
proxy_handler = urllib2.ProxyHandler({'http':'121.40.108.76:8080'})
# Build a custom opener with the proxy configuration
custom_opener = urllib2.build_opener(proxy_handler)
# Add a spoofed User-Agent to avoid detection
custom_opener.addheaders = [('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36')]
# Set the custom opener as the default handler
urllib2.install_opener(custom_opener)
response = urllib2.urlopen(test_url)

print(response.read().decode('gbk'))

To handle dead proxies, you can build a proxy pool and randomly select a proxy for each request:

# -*- coding: utf-8 -*-
import urllib2
import random

test_url = "http://www.ip181.com/"
# Predefined pool of working proxy addresses
proxy_pool = ['119.6.136.122:80','114.106.77.14:8080']
# Randomly select one proxy from the pool
selected_proxy = random.choice(proxy_pool)
proxy_handler = urllib2.ProxyHandler({'http': selected_proxy})

custom_opener = urllib2.build_opener(proxy_handler)
custom_opener.addheaders = [('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36')]

urllib2.install_opener(custom_opener)
response = urllib2.urlopen(test_url)

print(response.read().decode('gbk'))

You can build your own proxy pool by scraping public proxy listing sites, filtering out non-working proxies, and dynamically removing dead proxies from the pool over time.

Avoid Hidden Honeypot Traps

Many websites plant invisible links and input fields as honeypots to detect crawlers. If your crawler interacts with these hidden elements, your IP will be blocked immediately. Always check if an element is visible before interacting with it, as shown in this Selenium example:

from selenium import webdriver

target_url = 'http://pythonscraping.com/pages/itsatrap.html'
driver = webdriver.PhantomJS(executable_path="phantomjs.exe")
driver.get(target_url)

all_links = driver.find_elements_by_tag_name("a")
for link in all_links:
    if not link.is_displayed():
        print(f"The link {link.get_attribute('href')} is a trap")

all_inputs = driver.find_elements_by_tag_name("input")
for field in all_inputs:
    if not field.is_displayed():
        print(f"Do not change value of {field.get_attribute('name')}")

For the example trap page, this will output all hidden elements to avoid:

the link http://pythonscraping.com/dontgohere is a trap
do not change value of phone
do not change value of email

Distributed Crawling for Large Scale Projects

For large crawling projects, distributed crawling spreads the request load across multiple servers and IP addresses, reducing the risk of any single IP being blocked. A common distributed architecture built with Python, Scrapy and Redis follows this structure:

1. Use a mature HTTP crawling framework like Scrapy for core request handling
2. Use a Bloom Filter to avoid re-crawling duplicate pages
3. Maintain a shared distributed request queue accessible to all worker nodes
4. Integrate the distributed queue with Scrapy's crawling workflow
5. Add post-processing: content extraction with libraries like python-goose, and store results in MongoDB.

Simulate Human Login

For sites that require authentication, Selenium can be used with a regular browser like Firefox or a headless browser like PhantomJS to simulate a full human login session. This approach is far less likely to trigger anti-bot detection than raw request-based authentication.

Important Notes

1. The example proxy IPs provided are for demonstration only. You will need to source fresh working proxies for your own projects, as public proxies become inactive quickly.
2. The most commonly used effective combination for genarel crawling is spoofed request headers + rotating proxies. For JavaScript-heavy sites that do not return full content with simple request libraries, use Selenium paired with a headless or regular browser to get complete content.