Common Anti-Blocking Techniques for Python Web Scraping
Throttle Request Frequency
Many websites flag crawlers that send requests at super-human speeds, such as non-stop page crawling, rapid form submissions, or bulk image downloading. Two common approaches to add natural browsing delays are fixed waiting and conditinoal waiting.
Fixed Delay
This forces the crawler to pause for a set interval betwee ...
Posted on Sun, 27 Sep 2026 16:43:36 +0000 by payjo
Python Crawler for Downloading CSDN Personal Blog Content
This crawler fetches content from a CSDN personal blog (either the main page or a category) and saves each article as a text file. Ensure the target URL includes ?viewmode=contents to retrieev all entries.
Basic Crawler
The basic version extracts article links from a page and saves each article’s content:
# -*- coding: utf-8 -*-
import urllib2
...
Posted on Wed, 15 Jul 2026 16:55:41 +0000 by Dilb