Anti-Crawling Processing

Crawler: Using technical methods to batch obtain data from the internet

Anti-crawling: Using technical methods to prevent others from obtaining data from your website

Anti-anti-crawling: Using technical methods to bypass the opponent's anti-crawling strategies

Request Header Anti-crawling

User-Agent: A field in the HTTP request header that indicates what client is making the request. Different browsers have different User-Agents. Previously, we handled this anti-crawling method by directly copying from the browser, which was cumbersome. Python has a third-party module that helps us obtain User-Agents more quick: fake_useragent.

import requests
# pip install fake_useragent
from fake_useragent import UserAgent

# Randomly generate a UA, may have browser version compatibility issues
ua_generator = UserAgent()
request_headers = {
    'User-Agent': ua_generator.random
}
response = requests.get('https://search.jd.com/Search?keyword=%E6%89%8B%E6%9C%BA', headers=request_headers)
print(response.text)

Referer: Indicates which URL the current request came from. This can also be used as an anti-crawling technique. If the request doesn't come from the specified page, no relevant response is provided.

import requests
from fake_useragent import UserAgent
from lxml import etree

# 1. Send request to the list page
target_url = 'https://www.pearvideo.com/popular'
headers_config = {
    "user-agent": UserAgent().random
}
response_data = requests.get(target_url, headers=headers_config)
page_html = response_data.text
# 2. Parse detail page URLs from the list page HTML
parsed_html = etree.HTML(page_html)

video_items = html.xpath('//ul[@id="popularList"]/li')
for item in video_items:
    # 3. We find that the detail page doesn't contain the data we need
    detail_path = item.xpath('./div[@class="popularem-ath"]/a/@href')[0]
    video_title = item.xpath('./div[@class="popularem-ath"]/a/h2/text()')[0]
    content_id = detail_path.split('_')[1]
    # 4. The video URL is obtained from this link
    # 4.1 When we send a request to this link, we can't get the data we need
    # 4.2 Because the backend checks the Referer field in our request headers
    # Anti-leeching indicates the URL we were on before sending this request
    json_endpoint = f"https://www.pearvideo.com/videoStatus.jsp?contId={content_id}&mrd=0.4479471535100221"
    full_detail_url = f'https://www.pearvideo.com/{detail_path}'
    detail_headers = {
        "user-agent": UserAgent().random,
        "Referer": full_detail_url
    }
    json_response = requests.get(json_endpoint, headers=detail_headers).json()
    timestamp = json_response['systemTime']
    video_source = json_response['videoInfo']['videos']['srcUrl']
    final_video_url = video_source.replace(timestamp, f'cont-{content_id}')
    print(video_title, final_video_url)
    with open(f'{video_title}.mp4', 'wb') as file_handle:
        file_handle.write(requests.get(final_video_url).content)
    print(f'{video_title}.mp4 - Download completed')
    print('-' * 100)

Cookie: Corresponds to user information and has an expiration time. When encountered, we can simply replace the cookie value.