Core Concepts and Architecture

Nginx is a high-performance HTTP server and reverse proxy developed by Igor Sysoev. It efficiently handles concurrent connections and serves as a reliable alternative to traditional web servers under heavy load. Beyond web serving, it functions as an IMAP/POP3/SMTP proxy, making it a versatile tool for modern infrastructure.

Proxy Modes

Forward Proxy acts as an intermediary for clients accessing external resources. Reverse Proxy manages inbound traffic, distributing requests from external clients to internal backend services. The key distinction lies in traffic direction: forward proxies represent clients to the internet, while reverse proxies represent servers to clients.

Load Balancing Fundamentals

Load balancing divides into hardware-based solutions (e.g., F5 appliances) and software-based approaches like Nginx. Hardware balancers offer enterprise-grade stability at premium costs, whereas software solutions provide flexible, cost-effective traffic distribution.

The mechanism distributes incoming requests across server pools. For instance, with three backend servers processing nine requests using round-robin strategy, each server handles three requests, preventing overload on any single node.

Clustering with Tomcat and Redis

Implement session sharing across Tomcat instances using Redis. Deploy the session manager JAR to tomcat/lib and configure Redis endpoints in tomcat/conf.

Nginx Configuration for Tomcat Cluster

worker_processes  8;

events {
    worker_connections  2048;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile      on;
    keepalive_timeout  75;

    upstream backend_pool {
        server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
        server 10.0.1.10:8081 max_fails=3 fail_timeout=30s;
        server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 80;
        server_name api.example.com;

        location / {
            proxy_pass http://backend_pool;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 5s;
            proxy_send_timeout 10s;
            proxy_read_timeout 10s;
        }

        location ~ \.(jsp|do|action)$ {
            proxy_pass http://backend_pool;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root /usr/share/nginx/html;
        }
    }
}

Management Commands:

# Validate configuration
nginx -t -c /etc/nginx/nginx.conf

# Start service
nginx -c /etc/nginx/nginx.conf

# Reload without downtime
nginx -s reload

File Server Implementation

Configure Nginx to serve static assets directly. Key considerations include:

• Setting appropriate client_max_body_size for uploads
• Resolving permisssion issues by ensuring nginx worker processes can read files
• Implementing bandwidth throttling for large media files when network capacity is limited

Capturing Real Client IP Addresses

When operating behind proxies or CDNs, use the real_ip module to restore original client addresses.

http {
    log_format detailed '$msec | $time_local | $host | $status | $bytes_sent | $realip_remote_addr | $upstream_addr | $request | $request_time | $upstream_response_time';
    
    access_log /var/log/nginx/access.log detailed;

    # Trust headers from known proxies
    set_real_ip_from 10.0.0.0/8;
    set_real_ip_from 172.16.0.0/12;
    set_real_ip_from 192.168.0.0/16;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    # Forward IP headers to upstreams
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Trafffic Control and Rate Limiting

Connection Limiting

Define connection zones to restrict simultaneous connections:

http {
    limit_conn_zone $binary_remote_addr zone=addr_zone:20m;
    limit_conn_zone $server_name zone=svr_zone:20m;

    server {
        limit_conn addr_zone 15;    # Max 15 connections per IP
        limit_conn svr_zone 200;    # Max 200 connections per server
    }
}

Request Rate Limiting

Prevent abuse and DDoS attacks by throttling request rates:

http {
    limit_req_zone $binary_remote_addr zone=req_limiter:20m rate=5r/s;
    limit_req_zone $server_name zone=svr_limiter:20m rate=20r/s;

    server {
        # 5 req/sec per IP, burst up to 10
        limit_req zone=req_limiter burst=10 nodelay;
        
        # 20 req/sec per server, burst up to 20
        limit_req zone=svr_limiter burst=20;

        location /api/search {
            # Stricter limit for search endpoints
            limit_req zone=req_limiter burst=5;
        }
    }
}

Bandwidth Throttling

Control download speeds for resource-intensive content:

http {
    limit_conn_zone $binary_remote_addr zone=dl_zone:10m;

    server {
        location /downloads/ {
            limit_conn dl_zone 2;          # Max 2 download connections per IP
            limit_rate_after 1m;           # Apply limit after 1MB
            limit_rate 100k;               # Cap at 100KB/s
        }
    }
}

Comprehensive Rate Limiting Example

http {
    # Whitelist management
    geo $whitelist {
        default 0;
        10.0.0.0/8 1;
        192.168.1.0/24 1;
    }

    map $whitelist $limit_key {
        0 $binary_remote_addr;
        1 "";
    }

    limit_conn_zone $limit_key zone=conn_per_ip:15m;
    limit_req_zone $limit_key zone=req_per_ip:15m rate=3r/s;
    limit_req_zone $server_name zone=req_per_server:15m rate=30r/s;

    server {
        limit_req zone=req_per_server burst=15;
        limit_conn conn_per_ip 10;
        limit_req zone=req_per_ip burst=5;

        location /downloads/ {
            limit_conn conn_per_ip 1;
            limit_rate_after 2m;
            limit_rate 80k;
        }
    }
}

Important: Use $binary_remote_addr instead of $remote_addr for memory efficiency. The binary representation consumes 4 bytes for IPv4 and 16 bytes for IPv6, enabling approximately 160,000 tracking entries per 10MB of shared memory.

Multi-Port and TLS Configuration

Serving Multiple Ports

server {
    listen 8080;
    server_name app.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 8443 ssl http2;
    server_name app.example.com;

    ssl_certificate /etc/ssl/certs/app.crt;
    ssl_certificate_key /etc/ssl/private/app.key;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 60m;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
    ssl_prefer_server_ciphers off;

    location / {
        root /var/www/app;
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Multiple TLS Virtual Hosts

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate /etc/ssl/certs/api.crt;
    ssl_certificate_key /etc/ssl/private/api.key;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 60m;

    location / {
        proxy_pass http://backend_cluster;
    }
}

server {
    listen 443 ssl http2;
    server_name dashboard.example.com;

    ssl_certificate /etc/ssl/certs/dashboard.crt;
    ssl_certificate_key /etc/ssl/private/dashboard.key;

    location / {
        root /var/www/dashboard;
    }
}

Common Issues and Solutions

Missing Dynamic Libraries

Error: error while loading shared libraries: libssl.so.1.1

Solution: Create symbolic links or install missing dependencies:

ln -s /usr/local/lib/libssl.so.1.1 /lib/x86_64-linux-gnu/

Redirect Failures After Load Balancing

Symptom: Pages fail to redirect or show ERR_NAME_NOT_RESOLVED

Cause: Non-standard listening ports require explicit port headers

Solution: Append port to Host header:

location / {
    proxy_set_header Host $host:8080;
    proxy_pass http://backend_pool;
}

File Server Permission Denied

Symptom: HTTP 403 errors when serving static files

Root Cause: File ownership mismatch between nginx worker user and file creator

Solutions:

1. Adjust nginx user: user www-data; in nginx.conf
2. Modify file permissions: chown -R www-data:www-data /var/www/static
3. Add ACL entries: setfacl -R -m u:www-data:r-x /var/www/static