Implementing GPU Monitoring with NVIDIA GPU Exporter and Prometheus

Deploying NVIDIA GPU Exporter on GPU Servers

1. Acquire NVIDIA GPU Exporter

wget https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v1.2.0/nvidia_gpu_exporter_1.2.0_linux_x86_64.tar.gz

2. Launch NVIDIA GPU Exporter

tar xf nvidia_gpu_exporter_1.2.0_linux_x86_64.tar.gz
mkdir -p /opt/gpu-monitoring
mv nvidia_gpu_exporter /opt/gpu-monitoring/
nohup /opt/gpu-monitoring/nvidia_gpu_exporter > /var/log/gpu-exporter.log 2>&1 &

3. Verify Monitoring Data Collection

The service automatically listens on port 9835. Use curl to confirm that monitoring data is being properly collected:

curl http://localhost:9835/metrics

Integrating with Prometheus Monitoring

Add the following configuration to your prometheus.yml file to include the GPU exporter:

scrape_configs:
  - job_name: 'gpu-monitoring'
    static_configs:
      - targets: ['192.168.2.23:9830']
        labels:
          gpu_model: nvidia-rtx-4090
          location: datacenter-1
      - targets: ['192.168.2.26:9830']
        labels:
          gpu_model: nvidia-rtx-4080
          location: datacenter-1

Essential Monitoring Metrics

Metric Name	Description
gpu_utilization_percent	GPU utilization percentage
memory_used_bytes	Currently allocated GPU memory
memory_total_bytes	Total GPU memory capacity
gpu_temperature_celsius	GPU temperature in Celsius

Sample Metrics Interface Output

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 7
# HELP gpu_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which gpu_exporter was built.
# TYPE gpu_exporter_build_info gauge
gpu_exporter_build_info{branch="",goversion="go1.16.5",revision="",version=""} 1
# HELP gpu_utilization_percent Current GPU utilization percentage
# TYPE gpu_utilization_percent gauge
gpu_utilization_percent{uuid="df6e7a7c-7314-46f8-abc4-b88b36f3aa"} 0
# HELP memory_used_bytes Currently allocated GPU memory in bytes
# TYPE memory_used_bytes gauge
memory_used_bytes{uuid="df6e7a7c-7314-46f8-abc4-b88b36f3aa"} 7.06740224e+08
# HELP memory_total_bytes Total GPU memory capacity in bytes
# TYPE memory_total_bytes gauge
memory_total_bytes{uuid="df6e7a7c-7314-46f8-abc4-b88b36f3aa"} 8.589934592e+09
# HELP gpu_temperature_celsius Current GPU temperature in Celsius
# TYPE gpu_temperature_celsius gauge
gpu_temperature_celsius{uuid="df6e7a7c-7314-46f8-abc4-b88b36f3aa"} 34
# HELP power_consumption_watts Current power draw in watts
# TYPE power_consumption_watts gauge
power_consumption_watts{uuid="df6e7a7c-7314-46f8-abc4-b88b36f3aa"} 28.07