Web Scraping Strategies for Douban Movie Top 250
Data Extraction Using XPath and lxml
Automated data collection involves simulating user navigation to retrieve and organize information from web pages. The lxml library, combined with XPath, provides a high-performance way to traverse the DOM tree. The following implementation uses a persistent session and targets specific elements within the D ...
Posted on Mon, 02 Nov 2026 16:40:38 +0000 by typo
Resolving Python Requests Double-Encoding in CVE-2024-32709 Exploit
Vulnerability OverviewCVE-2024-32709 identifies a critical SQL Injection vulnerability in the WordPress WP-Recall plugin (versions <= 16.26.5). Unauthenticated attackers can exploit the group-name parameter within the account management endpoint to extract sensitive database records. The FOFA search query for identifying vulnerable deploymen ...
Posted on Sun, 01 Nov 2026 16:14:38 +0000 by webshifter
Automated Web Interactions: Leveraging Python’s Requests and BeautifulSoup Libraries
Library Installation
pip install requests beautifulsoup4 lxml
The lxml parser is recommended for performance and robustness when processing complex HTML structures.
Core HTML Parsing with BeautifulSoup
BeautifulSoup transforms raw markup into a navigable tree structure, enabling precise element extraction and manipulation.
from bs4 import Beau ...
Posted on Thu, 15 Oct 2026 16:26:49 +0000 by subhuman
Overcoming Challenges: Uploading WeChat Temporary Media with Python Requests
Project Goal
When a user uploads an image, the system receives it, performs array operations, and returns the processed image—all entirely in memory without touching the file system.
Project Analysis
WeChat's official upload API uses the following curl command:
curl -F media=@{image_name} "https://api.weixin.qq.com/cgi-bin/media/upload?acc ...
Posted on Mon, 28 Sep 2026 16:20:27 +0000 by dawieharmse
Python HTTP Libraries: urllib, urllib2, and requests Comparison
Understanding urllib and urllib2 Differences
Both urllib and urllib2 modules handle URL operations, but they serve different purposes. The key distinctions include:
urllib2.urlopen can accept either a Request object or a URL, while urllib.urlopen only accepts URLs
When using a Request object with urllib2, you can set custom headers for t ...
Posted on Fri, 25 Sep 2026 16:54:16 +0000 by myanavrin
Anti-Crawling Techniques: Request Header Manipulation
Anti-Crawling Processing
Crawler: Using technical methods to batch obtain data from the internet
Anti-crawling: Using technical methods to prevent others from obtaining data from your website
Anti-anti-crawling: Using technical methods to bypass the opponent's anti-crawling strategies
Request Header Anti-crawling
User-Agent: A field in the HTTP ...
Posted on Wed, 23 Sep 2026 16:06:09 +0000 by goldbug
Essential Python Libraries for PyCharm Development
PyCharm Overview
PyCharm is a comprehensive Python IDE offering features like intelligent code assistance, debugging tools, testing frameworks, and database integration. It supports development across various domains including web applications, data science, and machine learning.
HTTP Requests Library
The Requests library simplifies HTTP commun ...
Posted on Wed, 26 Aug 2026 16:21:53 +0000 by mcatalf0221
Building API Automation Test Framework with Python Requests
This framework implements automated interface testing using Python's unittest module combined with the requests library for HTTP operations.
Project Structure
aototest/
├── Common/ # Utility classes: database helpers, file operations
├── Config/ # Configuration: environment URLs, credentials, DB connections
├── Data/ ...
Posted on Sun, 19 Jul 2026 16:59:10 +0000 by businessman332211
Simulating GitHub Login with Python requests Library
Simulating GitHub Login
Login Process Overview
When implementing login simulation for GitHub, three main steps are required:
Retrieve the authenticity token from the login page using session.get
Construct form data and headers, then submit to the /session endpoint via session.post
Verify login status by parsing the profile page title
Main Fun ...
Posted on Sat, 11 Jul 2026 16:55:50 +0000 by Colton.Wagner
Advanced Web Scraping with Python Requests: Session Management, Proxies, and Thread Pools
Session-Based Cookie Handling
When scraping user-specific data, traditional requests.get() calls often fail to retrieve the target information. Consider a scenario where you need to access a user's profile page on a social networking site. Without proper cookie management, you'll receive the login page instead of the authenticated profile data. ...
Posted on Fri, 26 Jun 2026 17:27:49 +0000 by Genesis730