Overview of Prometheus

Prometheus is an open-source monitoring and alerting toolkit developed in Go. It excels at monitoring containerized environments, particularly gaining prominence alongside Kubernetes adoption. The system combines metric collection, storage, and querying capabilities into a unified solution.

Time Series Data Characteristics

Time Series Data refers to data points collected sequantially over time, capturing system or device state changes chronologically.

Key Advantages

• Superior Performance: Traditional relational databases struggle with large-scale data processing. While NoSQL solutions handle big data better, time series databases outperform both approaches.
• Cost Efficiency: Advanced compression algorithms minimize storage requirements and reduce I/O overhead effectively.

Prometheus employs highly efficient time series storage mechanisms. Each sample consumes approximately 3.5 bytes, allowing millions of time series with 30-second intervals over 60 days to occupy roughly 200GB (based on official benchmarks).

Core Features

• Multi-dimensional data model
• Flexible query language
• Independent single-node operation without distributed storage dependencies
• HTTP-based pull model for data collection
• Gateway support for push model scenarios
• Service discovery and static configuration for target identification
• Comprehensive visualization and dashboard support

Installing Prometheus Server

Visit https://prometheus.io/ for official resources.

Step 1: Download Binary Package

Obtain the precompiled binary package from the official repository:

[root@monitoring-server ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.25.0/prometheus-2.25.0.linux-amd64.tar.gz
[root@monitoring-server ~]# tar -xf prometheus-2.25.0.linux-amd64.tar.gz 
[root@monitoring-server ~]# ll
total 64732
-rw-------. 1 root root     1426 Nov 17 16:08 anaconda-ks.cfg
drwxr-xr-x  4 3434 3434      132 Feb 18 00:13 prometheus-2.25.0.linux-amd64
-rw-r--r--  1 root root 66280932 Feb 18 00:16 prometheus-2.25.0.linux-amd64.tar.gz
[root@monitoring-server ~]# mv prometheus-2.25.0.linux-amd64 /usr/local/prometheus-2-25.0

Step 2: Create Symbolic Link

[root@monitoring-server local]# ln -s /usr/local/prometheus-2-25.0 /usr/local/prometheus

Step 3: Configure Environment Variables

[root@monitoring-server ~]# vim /etc/profile
... ...
export PROMETHEUS_HOME=/usr/local/prometheus
PATH=$PATH:$PROMETHEUS_HOME
export PATH

source /etc/profile

Step 4: Configuration File Setup

[root@monitoring-server prometheus]# cat /usr/local/prometheus/prometheus.yml
# Collection frequency settings
global:
  scrape_interval:     15s # Collect metrics every 15 seconds. Default: 1 minute
  evaluation_interval: 15s # Evaluate rules every 15 seconds. Default: 1 minute
  # scrape_timeout uses global default (10s)

# Alerting configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Rule loading configuration
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# Target configuration
scrape_configs:
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

Step 5: Launch Service

[root@monitoring-server prometheus]# prometheus \
    --config.file=#[specify config file] \
    --web.listen-address="0.0.0.0:9090"#[default listening port]

[root@monitoring-server prometheus]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml"

Step 6: Access Web Interface

Navigate to http://your-server-ip:9090 in your browser.

Monitoring Remote Hosts

Step 1: Deploy Node Exporter on Target Host

# On the target server
[root@target-host ~]# cd /opt
[root@target-host opt]# wget https://github.com/prometheus/node_exporter/releases/download/v1.1.1/node_exporter-1.1.1.linux-amd64.tar.gz

Step 2: Extract Node Exporter

[root@target-host opt]# tar -xf node_exporter-1.1.1.linux-amd64.tar.gz -C /usr/local/
[root@target-host opt]# cd /usr/local/

Step 3: Run Node Exporter

[root@target-host ~]# nohup /usr/local/node_exporter-1.1.1.linux-amd64/node_exporter &

Step 4: Update Prometheus Configuration

[root@monitoring-server prometheus]# vim /usr/local/prometheus/prometheus.yml
# Add the following configuration
  - job_name: "remote-target"  # Name your monitoring target
    static_configs:
    - targets: ['192.168.15.31:9100']  # Specify IP and port
    - targets: #[multiple targets can be added]

Step 5: Verify Access

Check the Prometheus web interface to confirm target discovery.

MySQL Database Monitoring

Step 1: Deploy MySQL Instance

Using Kubernetes deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: mysql-deployment
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql-container
          image: mysql:5.7
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "123456"
---
kind: Service
apiVersion: v1
metadata:
  name: mysql-service
spec:
  ports:
    - port: 3306
      targetPort: 3306
      nodePort: 30306
  selector:
    app: mysql
  type: NodePort

Step 2: Install MySQL Exporter

[root@target-host opt]# tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz
[root@target-host opt]# mv mysqld_exporter-0.12.1.linux-amd64 mysql_exporter_0.12.1
[root@target-host opt]# mv mysql_exporter_0.12.1 /usr/local/
[root@target-host ~]# ll /usr/local/mysql_exporter_0.12.1/
total 14484
-rw-r--r-- 1 3434 3434    11325 Jul 29  2019 LICENSE
-rwxr-xr-x 1 3434 3434 14813452 Jul 29  2019 mysqld_exporter
-rw-r--r-- 1 3434 3434       65 Jul 29  2019 NOTICE

Step 3: Create Monitoring User in MySQL

[root@target-host ~]# kubectl exec -it mysql-pod-name -- bash
root@mysql-pod:/# mysql -p
Enter password:123456
mysql> GRANT SELECT,REPLICATION CLIENT,PROCESS ON *.* TO 'monitor_user'@'%' IDENTIFIED BY 'secure_password';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

Step 4: Create Connnection Configuration

[root@target-host mysql_exporter_0.12.1]# pwd
/usr/local/mysql_exporter_0.12.1
[root@target-host mysql_exporter_0.12.1]# vim .mysql.cnf
[client]
user=monitor_user
password=secure_password
port=30306

Step 5: Start MySQL Monitoring

[root@target-host mysql_exporter_0.12.1]# ./mysqld_exporter --config.my-cnf=".mysql.cnf"
INFO[0000] Starting mysqld_exporter (version=0.12.1, branch=HEAD, revision=48667bf7c3b438b5e93b259f3d17b70a7c9aff96)  source="mysqld_exporter.go:257"
INFO[0000] Build context (go=go1.12.7, user=root@0b3e56a7bc0a, date=20190729-12:35:58)  source="mysqld_exporter.go:258"
INFO[0000] Enabled collectors:                           source="mysqld_exporter.go:269"
INFO[0000]  --collect.global_status                      source="mysqld_exporter.go:273"
INFO[0000]  --collect.global_variables                   source="mysqld_exporter.go:273"
INFO[0000]  --collect.slave_status                       source="mysqld_exporter.go:273"
INFO[0000]  --collect.info_schema.innodb_cmp             source="mysqld_exporter.go:273"
INFO[0000]  --collect.info_schema.innodb_cmpmem          source="mysqld_exporter.go:273"
INFO[0000]  --collect.info_schema.query_response_time    source="mysqld_exporter.go:273"
INFO[0000] Listening on :9104

Step 6: Configure Prometheus for MySQL

[root@monitoring-server prometheus]# vim prometheus.yml
... ...
  - job_name: 'mysql-database'
    static_configs:
    - targets: ['192.168.15.31:9104']

Step 7: Verify MySQL Monitoring

Access the Prometheus web interface to confirm MySQL metrics collection.