Prometheus Overview

Prometheus is an open-source systems monitoring and alerting toolkit designed for reliability and scalability. Originally developed at SoundCloud and now a CNCF project, it excels at recording numeric time-series data.

Core Architecture

• Prometheus Server: The central component that scrapes and stores time-series data. It operates as a local time-series database and provides the PromQL query language for data analysis.
• Exporters: Libraries or standalone processes that expose metrics from third-party systems (e.g., MySQL, Redis) or custom applications in a format Prometheus can ingest.
• Pushgateway: A service that allows ephemeral or batch jobs to push their metrics to Prometheus, as the server typicaly operates on a pull-based model.
• Alertmanager: Manages alerts sent by client applications, handling deduplication, grouping, and routing to notification channels like email, Slack, or webhooks.
• Service Discovery: Prometheus supports various mechanisms (Consul, DNS, Kubernetes APIs, or file-based) to automatically discover scraping targets, reducing manual configuration overhead.

Deployment Guide

Prometheus Server Setup

1. Extract binary: tar -xzf prometheus-*.tar.gz -C /opt/prometheus/
2. Configure: Edit prometheus.yml to define scrape targets and alerting rules:

global:
  scrape_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

3. Launch: Run ./prometheus --config.file=prometheus.yml

Grafana Integration

Grafana connects to Prometheus as a data source to create visualization dashboards. After installing, add a new Prometheus data source using the server URL (e.g., http://localhost:9090). You can then import community-contributed dashboards or build custom panels using PromQL queries.

Alerting Configuration

To enable alerting, define rules in a YAML file and reference it in your Prometheus configuraton.

Defining Alert Rules

Create alerts.yml to define conditions:

groups:
  - name: system_alerts
    rules:
      - alert: HighMemoryUsage
        expr: 100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) > 85
        for: 2m
        annotations:
          summary: "Instance {{ $labels.instance }} memory usage is high"

Alertmanager Routing

Configure alertmanager.yml to define where notifications are sent:

route:
  receiver: 'email-notifications'

receivers:
  - name: 'email-notifications'
    email_configs:
      - to: 'admin@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alerting@example.com'

Application Monitoring

For custom applications (e.g., Spring Boot), use the Micrometer library to expose metrics. Once the /actuator/prometheus endpoint is active, update prometheus.yml to include the target:

- job_name: 'spring-boot-app'
  metrics_path: '/actuator/prometheus'
  static_configs:
    - targets: ['app-server:8080']

Dynamic Discovery

Instead of static IP lists, use file_sd_configs for dynamic target management. Create a targets.json file:

[
  {
    "targets": ["10.0.0.1:9100"],
    "labels": {"job": "node-exporter"}
  }
]

Reference this in prometheus.yml under file_sd_configs, and Prometheus will automatical pick up changes to this file without requiring a service restart.