Monitoring Architecture Overview

Effective monitoring requires understanding the behavior of the components being observed. This includes grasping the underlying principles of CPU operation, system scheduling, and resource allocation.

Key Monitoring Layers:

• Hardware Layer: Physical server health, component failures, and environmental factors.
• Operating System Layer: Kernel-level metrics including CPU, memory, disk I/O, and process states.
• Application Layer: Service-specific performance and availability metrics.
• Business Layer: High-level indicators of service functionality and user experience.

Establishing Performance Baselines:
To define a failure state, one must first establish a baseline for normal operations. For instance, determining what constitutes "high load" depends on the core count and the nature of the workload (I/O bound vs. CPU bound).

Hardware Monitoring Implementation

Modern servers typically include Baseboard Management Controllers (BMCs) such as iDRAC (Dell), ILO (HP), or IMM (IBM). These controllers allow for out-of-band management independent of the operating system.

IPMI (Intelligent Platform Management Interface)

IPMI is the standard protocol for hardware monitoring. To utilize IPMI on a Linux system, the required packages and services must be installed.

# Install IPMI tools
sudo dnf install OpenIPMI ipmitool -y

# Enable and start the IPMI service
sudo systemctl enable ipmi
sudo systemctl start ipmi

# Verify installation
ipmitool help

IPMI can be accessed locally or remotely over the LAN. Remote configuration requires setting up a network interface on the BMC, accessible via a dedicated management port or a shared NIC.

SNMP (Simple Network Management Protocol)

SNMP is widely used for collecting data from network devices and servers. Below is the setup procedure for the Net-SNMP daemon on a Linux host.

# Install SNMP packages
sudo dnf install net-snmp net-snmp-utils -y

# Backup default configuration
sudo cp /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.bak

# Create a minimal configuration
sudo tee /etc/snmp/snmpd.conf > /dev/null <<eof dontlogtcpwrappersconnects="" enable="" eof="" public_monitoring="" rocommunity="" serverroom_01="" service="" snmp="" snmpd="" start="" sudo="" sysadmin="" syscontact="" syslocation="" systemctl="" the="" yes=""></eof>

You can verify connectivity using snmpwalk or snmpget commands with specific Object Identifiers (OIDs).

# Retrieve system uptime
snmpget -v2c -c public_monitoring 127.0.0.1 1.3.6.1.2.1.1.3.0

# Retrieve 1-minute load average
snmpwalk -v2c -c public_monitoring 127.0.0.1 1.3.6.1.4.1.2021.10.1.3.1

Essential SNMP OIDs Reference:

Operating System Monitoring

OS monitoring focuses on the four primary resources: CPU, Memory, Disk I/O, and Network.

CPU Metrics

• Context Switches: The frequecny with which the kernel saves the state of one process and loads another. High rates may indicate thrashing.
• Run Queue (Load): The average number of processes waiting for CPU time.
• Utilization: Breakdown of time spent in user space, system space, and idle.

Performance Baseline Rules:

• Run Queue length should ideally not exceed the number of CPU cores (e.g., load < 4 for a 4-core machine).
• User CPU usage: 65-70% is healthy; higher indicates saturation.
• System CPU usage: Should generally remain below 30%.
• Idle time: Should be > 0%; consistent 0% indicates a bottleneck.

Memory Management

Memory is managed in pages (typically 4KB). It is crucial to monitor not just free memory, but also available memory (memory that can be reclaimed quickly from caches/buffers) and swap activity.

• SI (Swap In): Memory read from swap disk to RAM.
• SO (Swap Out): Memory written from RAM to swap disk.

Frequent swapping indicates memory exhaustion.

Disk I/O

Monitoring disk performance involves tracking IOPS (Input/Output Operations Per Second) and throughput.

• Sequential I/O: Large, contiguous file access (e.g., media streaming).
• Random I/O: Small, non-contiguous access (e.g., databases).

Performance Profiling with Nmon

Nmon is a performance monitoring tool for Linux that records data to a file for later analysis.

# Make the binary executable (example binary name)
chmod +x nmon_x86_rhel

# Run data collection: 10 snapshots every 5 seconds
./nmon_x86_rhel -f -s 5 -c 10 -m /var/tmp/

# Output filename format: hostname_YYMMDD_HHMM.nmon

To visualize the data, use the nmon_analyser spreadsheet tool to load the generated .nmon file and generate graphs.

Application Monitoring Example: Nginx

Monitoring web servers like Nginx typically involves enabling a status module that exposes internal metrics.

Building Nginx with Status Module

# Install build dependencies
sudo dnf install gcc pcre-devel zlib-devel openssl-devel wget -y

# Download source
wget https://nginx.org/download/nginx-1.24.0.tar.gz
tar -xzf nginx-1.24.0.tar.gz
cd nginx-1.24.0

# Create service user
sudo useradd -r -s /sbin/nologin nginx

# Compile with stub_status module
./configure \
  --prefix=/etc/nginx \
  --sbin-path=/usr/sbin/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --user=nginx \
  --group=nginx \
  --with-http_ssl_module \
  --with-http_stub_status_module

make && sudo make install

Configuring the Status Endpoint

Edit the Nginx configuration file to expose metrics on a specific port.

server {
    listen       8080;
    server_name  localhost;

    location /nginx_status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }
}

Reload the configuration and access the endpoint.

sudo nginx -s reload
curl http://127.0.0.1:8080/nginx_status

Understanding the Metrics:

• Active connections: Current open client connections.
• Reading: Connections currently reading request headers.
• Writing: Connections currently writing response bodies to clients.
• Waiting: Idle connections waiting for new requests.