Introduction to Prometheus

Foreword

Prometheus is a popular open-source monitoring and alerting system. It is well-suited for recording any numeric time-series data, making it ideal for both machine-centric monitoring and dynamic service-oriented architectures. The project is independent and has a very active community.

Official Documentation: https://prometheus.io/docs/prometheus/latest/getting_started/

Prometheus Architecture

Prometheus operates through a series of steps:

1. Data Collection (Exporters): Prometheus periodically scrapes metrics from configured targets via HTTP requests. Targets can include applications, systems, or services.
2. Data Storage (Storage): Collected data is stored in Prometheus's local time-series database. Each time series is uniquely identified by a metric name and a set of key-value labels.
3. Data Querying (PromQL): Prometheus utilizes its query language, PromQL, to aggregate and retrieve specific metric data from the storage angine.
4. Alerting (Alertmanager): Prometheus can trigger alerts based on user-defined rules. When metrics exceed defined thresholds, Prometheus sends alerts to Alertmanager. Alertmanager handles grouping, deduplication, routing, and delivery of alerts to receivers like email or messaging platforms.
5. Visualization (Grafana): Tools like Grafana are commonly used to visualize Prometheus data, creating dashboards with graphs, logs, and alert statuses.

Time-Series Data in Prometheus

What is Time-Series Data?

Time-series data refers to data points recorded in chronological order, capturing the state changes of systems or devices over time.

Characteristics of Time-Series Data

• Performance: Time-series databases are optimized for handling large volumes of time-stamped data, significantly outperforming traditional relational databases in this regard.
• Storage Efficiency: Advanced compression algorithms minimize storage space and reduce I/O operations. Prometheus is known for its efficient storage, with each sample occupying approximately 3.5 bytes.

Prometheus Use Cases

Prometheus excels at monitoring any system that generates numeric time-series data. It is particularly effective for monitoring highly dynamic, service-oriented environments and infrastructure.

Deployment Configuration

A comprehensive monitoring stack often involves several components:

• Prometheus: The core monitoring service.
• node-exporter: Collects system-level metrics.
• kube-state-metrics: Exposes cluster-level metrics from the Kubernetes API.
• metrics-server: Provides resource usage metrics to pods and nodes.
• Consul: For service discovery.
• blackbox exporter: For black-box probing of endpoints.
• Alertmanager: Handles alert routing and management.
• Grafana: For data visualization.
• prometheusAlert: A service for forwarding alerts.

Deploying Prometheus

This section details the steps to deploy an externally accessible Prometheus instance within a Kubernetes cluster.

1. Create Namespace

First, create a dedicated namespace for Prometheus.

kubectl create namespace monitor

2. Create RBAC Rules

Define necessary Role-Based Access Control (RBAC) configurations, including a ServiceAccount, ClusterRole, and ClusterRoleBinding.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources: ["nodes","nodes/proxy","services","endpoints","pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources: ["ingress"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitor

Verify the creation of these resources:

kubectl get sa prometheus -n monitor
kubectl get clusterrole prometheus
kubectl get clusterrolebinding prometheus

3. Create Prometheus Configuration ConfigMap

Create a ConfigMap to hold the main Prometheus configuration file (prometheus.yml).

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitor
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
      external_labels:
        cluster: "kubernetes"

    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: 'prometheus'

    rule_files:
    - '/etc/prometheus/rules/*.rules

Verify the ConfigMap:

kubectl get cm prometheus-config -n monitor

4. Create Prometheus Rules ConfigMap

Create another ConfigMap for Prometheus alert rules (general.rules and node.rules).

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rules
  namespace: monitor
data:
  general.rules: |
    groups:
    - name: general.rules
      rules:
      - alert: InstanceDown
        expr: |
          up{job=~"k8s-nodes|prometheus"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} host {{ $labels.hostname }} has been down for more than 1 minute."

  node.rules: |
    groups:
    - name: node.rules
      rules:
      - alert: NodeFilesystemUsage
        expr: |
          100 - (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }}: {{ $labels.mountpoint }} filesystem usage is high"
          description: "{{ $labels.instance }} host {{ $labels.hostname }}: {{ $labels.mountpoint }} filesystem usage is above 85% (current value: {{ $value }})."

Verify the rules ConfigMap:

kubectl get cm -n monitor prometheus-rules

5. Create Prometheus Service

Define a Kubernetes Service to expose Prometheus internally within the cluster.

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitor
  labels:
    k8s-app: prometheus
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 9090
    targetPort: 9090
  selector:
    k8s-app: prometheus

6. Create Prometheus PersistentVolumeClaim

To ensure data persistence across pod restarts, create a PersistentVolumeClaim (PVC). This example assumes a StorageClass named nfs-storage is available, possibly provisioned via NFS.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data-pvc
  namespace: monitor
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "nfs-storage"
  resources:
    requests:
      storage: 10Gi

7. Create Prometheus Deploymant

Create the Deployment resource for Prometheus. This includes the main Prometheus container and a configmap-reload container for dynamically updating configurations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitor
  labels:
    k8s-app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: prometheus
  template:
    metadata:
      labels:
        k8s-app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.36.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9090
        securityContext:
          runAsUser: 65534
          privileged: true
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--web.enable-lifecycle"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention.time=10d"
        - "--web.console.libraries=/etc/prometheus/console_libraries"
        - "--web.console.templates=/etc/prometheus/consoles"
        resources:
          limits:
            cpu: 2000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 9090
          initialDelaySeconds: 5
          timeoutSeconds: 10
        livenessProbe:
          httpGet:
            path: /-/healthy
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
        volumeMounts:
        - name: data
          mountPath: /prometheus
        - name: config
          mountPath: /etc/prometheus
        - name: prometheus-rules
          mountPath: /etc/prometheus/rules
      - name: configmap-reload
        image: jimmidyson/configmap-reload:v0.5.0
        imagePullPolicy: IfNotPresent
        args:
        - "--volume-dir=/etc/config"
        - "--webhook-url=http://localhost:9090/-/reload"
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
        volumeMounts:
        - name: config
          mountPath: /etc/config
          readOnly: true
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: prometheus-data-pvc
      - name: prometheus-rules
        configMap:
          name: prometheus-rules
      - name: config
        configMap:
          name: prometheus-config

Key parameters for the Prometheus container:

• --web.enable-lifecycle: Enables the /-/reload endpoint for configuration updates.
• --config.file: Path to the Prometheus configuration file within the container.
• --storage.tsdb.path: Directory for time-series data storage.
• --storage.tsdb.retention.time: Duration for retaining data (e.g., 10 days).
• --web.console.libraries and --web.console.templates: Paths for Prometheus's web console assets.

Verify the deployment and pods:

kubectl get deploy -n monitor
kubectl get pods -n monitor

8. Create Prometheus Ingress

Configure an Ingress resource to allow external access to Prometheus via a specific domain name.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: monitor
  name: prometheus-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: prometheus.k8s.cn
    http:
      paths:
        - pathType: Prefix
          backend:
            service:
              name: prometheus
              port:
                number: 9090
          path: /

Test external access:

curl prometheus.k8s.cn

You should receive a redirect indicating success, for example: <a href="/graph">Found</a>.