Overview of mitmproxy

mitmproxy is an interactive, SSL-capable man-in-the-middle proxy for HTTP and HTTPS traffic. It provides similar capabilities to graphical tools like Fiddler or Charles, but operates primarily within a terminal environment. The ecosystem includes three distinct interfaces:

• mitmproxy: A fully interactive terminal user interface for real-time traffic inspection.
• mitmweb: A web-based interface offering a visual dashboard to monitor intercepted requests.
• mitmdump: A command-line counterpart designed for headless operation and direct integration with custom Python scripts.

Core capabilities include intercepting and modifying requests/responses, replaying traffic, reverse proxying, and transparent proxying on macOS and Linux. The tool operates by binding to a local port (default 8080) on a host machine. When a client device on the same network configures its network proxy to point to this host, all web traffic routes through mitmproxy. This allows the proxy to inspect, alter, or log the data before forwarding it to the destination server, and vice versa for the response. By coupling this with Python scripts via mitmdump, developers can automate data extraction and persistence directly from the intercepted payloads.

Installation and Environment Configuration

Install the package using pip:

pip install mitmproxy

If network timeouts occur during download, extend the pip timeout threshold:

pip install mitmproxy --timeout 120

Note that the interactive terminal interface (mitmproxy) is not supported on Windows. However, Windows users can fully utilize mitmweb for a graphical experience and mitmdump for scripting and headless execution.

SSL Certificate Installation

To decrypt HTTPS traffic, the client device must trust the mitmproxy Certificate Authority (CA). Generating the CA certificates is achieved by simply launching the proxy once, which populates the ~/.mitmproxy directory with the necessary files.

Windows Configuration

Locate the mitmproxy-ca-p12.p12 file and double-click to initiate the Windows Certificate Import Wizard. Proceed through the default prompts without setting a password. When prompted for the certificate store location, manually select "Trusted Root Certification Authorities" to ensure all HTTPS connections are trusted without warnings. Confirm any security dialogs that appear.

Android Configuration

Android devices require the mitmproxy-ca-cert.pem file (or .cer if the system rejects .pem). Transfer the file to the phone and install it via the security settings, naming the credential appropriately.

An alternative approach avoids manual file transfers. First, identify the host machine's local IP address. Launch the web interface using mitmweb, which starts the proxy on port 8080 and the UI on port 8081. On the Android device, configure the Wi-Fi proxy settings to use the host machine's IP and port 8080. Once routed, open a browser on the mobile device and navigate to http://mitm.it/. This portal provides platform-specific certificate download links. After installation, navigate to Settings > Security > Encryption & Credentials, and explicitly enable full trust for the newly added root certificate.

Scripting Traffic Manipulation with mitmdump

mitmdump's primary advantage over traditional proxies is its seamless Python integration. Instead of manually analyzing packets, developers can define event hooks that automatically process traffic.

Capturing and Hooking Scripts

Raw traffic can be dumped directly to a file for offline analysis:

mitmdump -w captured_traffic.flow

To inject custom logic, attach a Python script using the -s flag:

mitmdump -s traffic_processor.py

Modifying Outgoing Requests

The request hook intercepts payloads before they reach the target server. For instance, spoofing the User-Agent header can be implemented as follows:

from mitmproxy import http

def request(packet):
    custom_agent = "Mozilla/5.0 (CustomBot; Linux)"
    packet.request.headers["User-Agent"] = custom_agent
    print(f"Headers updated: {packet.request.headers}")

Executing this script forces all routed client requests to present the custom User-Agent string, which is verified by inspecting the printed output on the host console.

Structured Console Logging

While standard print() works, the ctx module offers color-coded logging for better debugging visibility:

from mitmproxy import ctx, http

def request(packet):
    packet.request.headers["X-Intercepted"] = "true"
    ctx.log.info("Standard interception log.")
    ctx.log.warn("Potential issue detected in payload.")
    ctx.log.error("Critical failure or blocking response.")

These methods map to white, yellow, and red console output respectively, allowing developers to categorize diagnostic messages rapidly.

Extracting Request Metadata

Beyond headers, the HTTPFlow object exposes comprehensive request details. Accessing these attributes allows for conditional routing and filtering:

from mitmproxy import ctx, http

def request(packet):
    req_obj = packet.request
    logger = ctx.log.info
    
    logger(f"Target URL: {req_obj.url}")
    logger(f"HTTP Method: {req_obj.method}")
    logger(f"Destination Host: {req_obj.host}")
    logger(f"Connection Scheme: {req_obj.scheme}")

Attributes can also be overwritten on the fly to redirect traffic. Changing the destination URL forces the client to load content from an alternate endpoint:

from mitmproxy import http

def request(packet):
    redirect_destination = "https://httpbin.org/user-agent"
    packet.request.url = redirect_destination

Processing Server Responses

For data harvesting, the response hook provides access to the server's reply, including the crucial HTML or JSON payload:

from mitmproxy import ctx, http

def response(packet):
    server_reply = packet.response
    logger = ctx.log.info
    
    logger(f"Status Code: {server_reply.status_code}")
    logger(f"Response Length: {len(server_reply.text)} characters")
    
    if server_reply.status_code == 200:
        # Custom extraction logic can be applied here
        # e.g., parsing JSON, saving to a database
        pass