Web Protocols and HTTP Basics

HTTP communication relies on specific headers to manage content negotiation and client identification. Key request headers include Host, User-Agent, Content-Type, Accept-Encoding, Cookie, and Referer. The standard methods for interacting with resources are GET (retrieve), POST (create), PUT (update), DELETE (remove), PATCH (partial update), HEAD, OPTIONS, and TRACE.

Common status codes indicate the result of a request:

• 2xx (Success): 200 OK.
• 3xx (Redirection): 301 Moved Permanently, 302 Found (Temporary Redirect).
• 4xx (Client Error): 403 Forbidden, 404 Not Found.
• 5xx (Server Error): 500 Internal Server Error, 503 Service Unavailable.

The WebSocket protocol facilitates full-duplex communication over a single TCP connection. It begins as an HTTP request where the client sends an Upgrade: websocket header. The server acknowledges this upgrade, switching the connection from HTTP to the binary-framed WebSocket protocol.

Python Web Frameworks: Django, Flask, and Tornado

Django is a high-level framework with "batteries included," featuring an ORM, admin interface, caching, and middleware out of the box. Flask is a micro-framework focusing on simplicity and extensibility, relying on extensions like Flask-SQLAlchemy and Flask-Login. Tornado differs by providing asynchronous non-blocking I/O, making it suitable for long-polling and WebSocket applications. Both Django and Flask rely on the WSGI (Web Server Gateway Interface) standard, utilizing servers like Gunicorn or uWSGI to handle socket connections.

Django Architecture and Components

Request Lifecycle: A request enters via WSGI, passes through middleware (process_request), matches URL patterns, executes the corresponding view function (interacting with models and templates if necessary), and returns a response pass through middleware (process_response) back to the client.

FBV vs. CBV: Function-Based Views (FBV) are simple functions handling requests. Class-Based Views (CBV) utilize classes and inheritance. CBVs use a dispatch method to route requests to the appropriate method (e.g., get, post) based on reflection. This structure promotes code reuse but increases abstraction.

Forms and Models: Django Form classes handle user input validation and HTML rendering. ModelForm creates forms mapping directly to database models, reducing boilerplate code.

Middleware: Middleware classes act as hooks for processing requests and responses globally. Common use cases include authentication, CSRF protection, and request logging.

CSRF and AJAX Implementation

Django prevents Cross-Site Request Forgery by embedding a token in forms. For AJAX requests, this token can be retrieved from the cookie and added to the request header:

$.ajax({
    type: 'POST',
    url: '/api/endpoint',
    headers: { 'X-CSRFToken': $.cookie('csrftoken') },
    success: function(response) { console.log(response); }
});

Database Systems and Optimization

Indexes: Database indexing speeds up data retrieval. Types include standard indexes, unique indexes, and composite indexes. The "Leftmost Prefix Principle" dictates that for a composite index (A, B), queries using A or A AND B utilize the index, while queries using only B do not.

InnoDB vs. MyISAM: InnoDB supports transactions, row-level locking, and foreign keys, making it suitable for high-concurrency applications. MyISAM offers faster read speeds but only supports table-level locking and lacks transaction support.

Optimization Strategies: Techniques include avoiding SELECT *, optimizing WHERE clauses, creating appropriate indexes, using EXPLAIN for query analysis, separating reads from writes, and partitioning large tables.

Redis Caching and Data Structures

Redis is an in-memory data structure store. Unlike Memcached, Redis supports persistence, diverse data types (Strings, Lists, Hashes, Sets), and complex operations. Redis eviction policies (e.g., LRU, TTL-based) manage memory when limits are reached.

Connection Pooling in Python

import redis
from redis.connection import ConnectionPool

pool = ConnectionPool(host='localhost', port=6379, db=0)
client = redis.Redis(connection_pool=pool)
client.set('session_key', 'session_data')

Web Scraping with Scrapy

Scrapy is an asynchronous framework for web crawling. Its architecture consists of an Engine, Scheduler, Downloader, Spiders, and Item Pipelines. Middleware components allow for custom processing, such as setting proxies or handling retries.

class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta['proxy'] = 'http://proxy_ip:port'

Python Core Concepts

Reflection: Reflection allows inspecting and modifying an object's attributes at runtime using functions like getattr, setattr, and hasattr.

Garbage Collection: Python manages memory using reference counting and a generational garbage collector to handle reference cycles.

Concurrency: Threading is suitable for I/O-bound tasks, while multiprocessing is required for CPU-bound tasks due to the Global Interpreter Lock (GIL).

Singleton Pattern Implementation

import threading

class Singleton:
    _instance = None
    _lock = threading.Lock()

    def __new__(cls, *args, **kwargs):
        if cls._instance is None:
            with cls._lock:
                if cls._instance is None:
                    cls._instance = super().__new__(cls)
        return cls._instance

DevOps and Tools

Docker: Containerization tool. Common commands include docker build, docker run, and docker exec. Dockerfiles define the environment setup.

Ansible: An automation engine for configuration management. It uses YAML playbooks to define tasks executed on remote hosts via SSH.

Git: Version control system. Core workflow involves branching, committing, and merging. Commands like git rebase and git cherry-pick manage development history.