Working with the Requests Library

Example 1: Basic POST Request

import requests

# Construct POST payload
payload = {
    'username': 'admin',
    'password': 'secret123'
}

# Send POST request
target_url = 'https://api.example.com/login'
result = requests.post(target_url, data=payload)

A simple POST request typically includes a dictionary of key-value pairs sent as form data to the server.

Example 2: HTTP Basic Authentication

import requests

# Define authentication credentials
credentials = ('admin', 'securepass')

# Send request with auth header
target_url = 'https://internal.company.com/api/data'
result = requests.get(target_url, auth=credentials)

When accessing internal network resources that require authentication, pass a tuple containing username and password to the auth parameter.

Example 3: Using Proxy Servers

import requests

# Target URL
endpoint = 'https://httpbin.org/ip'
request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
}

# Define proxy configuration
proxy_config = {
    'http': '203.156.192.123:8080',
    'https': '203.156.192.123:8080'
}

# Make request through proxy
response = requests.get(url=endpoint, headers=request_headers, proxies=proxy_config)
print(f"Status: {response.status_code}")

Proxy servers help mask your real IP address and can bypass geographic restrictions on certain content.

Example 4: Handling SSL Certificate Issues

import requests

target_url = 'https://self-signed.cert-site.com/'
request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
}

# Standard request will fail with untrusted certificates
try:
    response = requests.get(url=target_url, headers=request_headers)
except Exception as e:
    print(f"Connection error: {e}")

Why this fails: HTTPS connections normally require verification by a trusted Certificate Authority (CA). Some websites, especially older or enterprise applications, use self-signed certificates that browsers and the requests library cannot verify automatically.

Correct Approach: Disabling SSL Verification

import requests

target_url = 'https://self-signed.cert-site.com/'
request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
}

# Bypass SSL certificate verification
response = requests.get(url=target_url, headers=request_headers, verify=False)
html_content = response.content.decode('utf-8')

# Save response to file
with open('response.html', 'w', encoding='utf-8') as output_file:
    output_file.write(html_content)

Setting verify=False tells requests to ignore certificate validation. Use this only when working with known internal systems where security implications are understood.

Example 5: Maintaining Sessions with Cookies

Demonstrated with https://example-login-site.com/

import requests

# Protected resource URL
protected_endpoint = 'https://example-login-site.com/dashboard'
request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
}

# Raw cookie string from browser inspection
raw_cookies = 'session_id=abc123def456; user_token=xyz789; remember_me=true'

# Parse cookie string into dictionary format
cookie_dict = {}
cookie_pairs = raw_cookies.split('; ')
for pair in cookie_pairs:
    key_value = pair.split('=', 1)
    cookie_dict[key_value[0]] = key_value[1]

# Access protected page with cookies
response = requests.get(url=protected_endpoint, headers=request_headers, cookies=cookie_dict)
page_content = response.content.decode('utf-8')

with open('authenticated_page.html', 'w', encoding='utf-8') as output_file:
    output_file.write(page_content)

After inspecting browser cookies, convert the semicolon-separated string into a dictionary that requests can use for authenticated requests.

Example 6: Session-Based Requests

import requests

# Create session object for cookie persistence
session = requests.Session()

# Define headers
request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
}

# Simulate login sequence
login_url = 'https://example-login-site.com/auth/login'
credentials = {'username': 'testuser', 'password': 'testpass'}
session.post(login_url, data=credentials, headers=request_headers)

# Subsequent requests automatically include stored cookies
dashboard_url = 'https://example-login-site.com/dashboard'
response = session.get(dashboard_url, headers=request_headers)

print(f"Dashboard accessible: {response.status_code == 200}")

Using a Session object automatically maintains cookies across multiple requests, mimicking browser behavior and simplifying authenticated workflows.

Data Parsing with Regular Expressions

Example 1: Greedy Matching

import re

# Greedy pattern: matches from first marker to last marker
sample_text = 'start_middle_section_with_lots_of_content_end'
regex_pattern = re.compile('start(.*)end')
matches = regex_pattern.findall(sample_text)
print(matches)

Output:

['middle_section_with_lots_of_content']

Greedy quantifiers (.*) match as much as possible, consuming everything between the first opening marker and the last closing marker.

Example 2: Non-Greedy (Lazy) Matching

import re

# Non-greedy pattern: stops at first occurrence
sample_text = 'start_first_marker_end_another_start_second_end'
regex_pattern = re.compile('start(.*?)end')
matches = regex_pattern.findall(sample_text)
print(matches)

Output:

['first_marker', 'second']

Adding ? after a quantifier makes it non-greedy, matching the minimum possible instead of maximum.

Example 3: Matching Across Newlines (Without Flag)

import re

# Sample text with multiple lines
multiline_text = """
    first_line_content
    second_line_ends_hereN
"""
regex_pattern = re.compile('first(.*)N')
matches = regex_pattern.findall(multiline_text)
print(f"Matches found: {len(matches)}")

Output:

By default, the . pattern does not match newline characters (\n), so content across lines remains unmatched.

Example 4: Matching Across Newlines (With re.S Flag)

import re

# Sample text with multiple lines
multiline_text = """
    first_line_content
    second_line_ends_hereN
"""
regex_pattern = re.compile('first(.*)N', re.S)
matches = regex_pattern.findall(multiline_text)
print(matches)

Output:

['_line_content\n    second_line_ends_here']

The re.S flag (also written as re.DOTALL) allows the dot to match newline characters, enabling multi-line content extraction.

Example 5: Case-Insensitive Matching

import re

multiline_text = """
    header_data_N
    footer_data_n
"""
# Combine DOTALL and IGNORECASE flags
regex_pattern = re.compile('header(.*)n', re.S | re.I)
matches = regex_pattern.findall(multiline_text)
print(matches)

Output:

['_data_\n    footer_data_']

Using re.I (or re.IGNORECASE) makes the pattern case-insensitive, matching both uppercase and lowercase characters.

Example 6: Matching Numeric Strings

import re

# Pattern for pure digit strings
digit_pattern = re.compile(r'^\d+$')
test_string = '4920173'

# match() checks from the beginning
validation = digit_pattern.match(test_string)

if validation:
    print(f"Extracted number: {validation.group()}")

Output:

Extracted number: 4920173

The \d metacharacter matches any single digit (0-9). Anchoring with ^ and $ ensures the entire string consists only of digits.

Example 7: Character Ranges

import re

numeric_string = '984372'
# Match digits 0-9: same as \d
pattern = re.compile('[0-9]')
matches = pattern.findall(numeric_string)
print(matches)

Output:

['9', '8', '4', '3', '7', '2']

Character classes using brackets allow custom matching sets. [1-5] would match only digits 1 through 5.

Regex Method Reference

match(): Validates from string start, returns match object or None search(): Locates first match anywhere in string findall(): Returns list of all non-overlapping matches sub(): Replaces matched portions with replacement string split(): Splits string at pattern boundaries