Setting Up Server Performance Monitoring

For effective server performance monitoring, implementing specialized tools becomes essential. Two primary approaches were evaluated:

• InfluxDB + Telegraf + Grafana
• Prometheus + Node Exporter + Grafana

The InfluxDB documentation appeared inconsistent across versions, making Prometheus the preferred choice.

Installation Process

# Install Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
tar -zxvf prometheus-2.52.0.linux-amd64.tar.gz

# Install Node Exporter
# Version: node_exporter-1.8.1.linux-amd64.tar.gz

# Install Grafana (avoid enterprise version)
wget https://dl.grafana.com/oss/release/grafana_11.0.0_amd64.deb
sudo dpkg -i grafana_11.0.0_amd64.deb

# Installation methods comparison:
# 1. Package managers (apt/yum/brew): Automatically install to system directories and register services
# 2. Manual package installation (dpkg): Similar to method 1, requires manual command execution
# 3. Binary file installation: Requires manual copying to system directories and service configuration

Configuring Systemd Services

# Grafana was installed via dpkg, automatically registered with systemd
systemctl start grafana-server  # Start Grafana
systemctl status grafana-server # Check Grafana status

# Copy binaries to system path
cp ${prometheus_directory}/prometheus /usr/local/bin
cp ${node_exporter_directory}/node_exporter /usr/local/bin

# Create Prometheus systemd service file
sudo vim /etc/systemd/system/metrics-collector.service

# Content for metrics-collector.service
#*********************************************************************
[Unit]
Description=Metrics Collector Service
Wants=network-online.target
After=network-online.target

[Service]
User=monitoring-user
Restart=on-failure

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/config.yml \
  --storage.tsdb.path=/var/lib/prometheus/data \
  --web.listen-address=:9095

[Install]
WantedBy=multi-user.target
#*********************************************************************

# Create Node Exporter service file
sudo vim /etc/systemd/system/system-exporter.service

# Content for system-exporter.service
#*********************************************************************
[Unit]
Description=System Metrics Exporter
After=network.target

[Service]
User=monitoring-user
ExecStart=/usr/local/bin/system-exporter

[Install]
WantedBy=multi-user.target
#*********************************************************************

# Reload systemd configuration
sudo systemctl daemon-reload
sudo systemctl start metrics-collector.service
sudo systemctl start system-exporter.service

# Enable services on boot
sudo systemctl enable metrics-collector.service
sudo systemctl enable system-exporter.service

# Monitor service logs
sudo journalctl -u metrics-collector.service -f
sudo journalctl -u system-exporter.service -f

Service Configuration Essentials

Key configurasion points:

• User and group in [Service] section must exist in the system
• Prometheus default port is 9090; use --web.listen-address for custom ports
• Configuration file location specified with --config.file parameter
• Targets in prometheus.yml should reference Node Exporter's port (default 9100)

Integrating with Grafana

Access Grafana at localhost:3000 using default credentials (admin/admin). Add Prometheus as a data source by navigating to the data sources section. After adding the data source, create a new dashboard. When metric options appear in the dropdown menu, the integration is successful. Remember to save all dashboard modifications manually.

Additional Commands

# Reset Grafana admin password
sudo grafana-cli admin reset-admin-password admin