Deploying the Node Exporter

To begin collecting system-level metrics, deploy the Node Exporter container. This requires mounting host paths to allow the exporter to read proc and sys filesystem statistics.

docker run -d \
  --name node-metrics \
  --restart unless-stopped \
  -p 9100:9100 \
  -v "/proc:/host/proc:ro" \
  -v "/sys:/host/sys:ro" \
  -v "/:/host/rootfs:ro" \
  prom/node-exporter

Verify collection by accessing the metrics endpoint at http://192.168.4.12:9100/metrics.

Configuring and Running Prometheus

Create a configuration file to define scrape targets. The example below configures monitoring for the Prometheus server itself and a Linux host.

cat > /etc/prometheus/config.yml <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus-core'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: 'server-01'

  - job_name: 'linux-hosts'
    static_configs:
      - targets: ['172.17.2.4:9100']
        labels:
          instance: 'node-01'
EOF

Launch the Prometheus server, mounting the configuration file created above.

docker run -d \
  --name prometheus-core \
  -v /etc/prometheus/config.yml:/etc/prometheus/prometheus.yml \
  --restart unless-stopped \
  -p 9090:9090 \
  prom/prometheus

Setting Up Grafana

Deploy Grafana to visualize the collected metrics. Ensure the designated data directory has write permissions.

mkdir -p /srv/grafana-data
chmod 777 /srv/grafana-data

docker run -d \
  --name grafana-ui \
  -v /srv/grafana-data:/var/lib/grafana \
  --restart unless-stopped \
  -p 3000:3000 \
  grafana/grafana

Monitoring Network Infrastructure via SNMP

For network devices like routers, the SNMP Exporter is required. This involves generating a specific configuration file using the device's MIBs.

1. Build Environment for Generator
Use a temporary container to install dependencies and build the configuration generator.

# Run a build environment
docker run -it --name build-env centos:7 bash

# Inside the container
yum install -y epel-release golang git make net-snmp net-snmp-utils net-snmp-devel
git clone https://github.com/prometheus/snmp_exporter.git /opt/snmp_exporter
cd /opt/snmp_exporter/generator
go build

2. Generate Configuration
Copy the necessary MIB files (e.g., Huawei) into the mibs directory and create a generator.yml.

# Example generator.yml
modules:
  huawei_switch:
    walk:
      - sysUpTimeInstance
      - sysDescr
      - 1.3.6.1.2.1.1
    version: 2
    auth:
      community: public
    lookups:
      - source_indexes: [ifIndex]
        lookup: ifDescr

Run the generator to produce snmp.yml:

export MIBDIRS=mibs
./generator generate

Copy the resulting snmp.yml to the host system.

3. Deploy SNMP Exporter

docker run -d \
  --name snmp-collector \
  -v /opt/snmp-config:/etc/snmp_exporter \
  --restart unless-stopped \
  -p 9116:9116 \
  prom/snmp-exporter

4. Update Prometheus Configuration
Add a scrape job for SNMP devices to the Prometheus configuration file. This uses relabeling to direct queries through the SNMP exporter.

  - job_name: 'network-snmp'
    scrape_interval: 120s
    static_configs:
      - targets:
        - 192.168.18.111
    metrics_path: /snmp
    params:
      module: [huawei_switch]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.17.2.5:9116  # SNMP Exporter IP

Restart the Prometheus container to apply changes: docker restart prometheus-core.

Inspecting Container Parameters

To retrieve the run command of an existing container, you can use utility tools.

# Method 1: Using runlike via pip
pip install runlike
runlike -p container_name

# Method 2: Using a helper container
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  cucker/get_command_4_run_container container_name