Monitoring Workflow for Non-Native Applications

Prometheus integration generally falls into two categories: applications that expose native metrics endpoints and those that do not. For legacy applications or services lacking a built-in /metrics interface, monitoring requires an intermediate component. The standard workflow involves the following stages:

1. Configure the target application to expose status data.
2. Deploy a Prometheus Exporter to scrape this data and translate it.
3. Create a Kubernetes Service to expose the Exporter.
4. Define a ServiceMonitor to instruct Prometheus to scrape the Exporter.

Case Study: Monitoring Nginx

To illustrate this process, we will configure monitoring for a standard Nginx deployment. Since Nginx does not provide Prometheus metrics by default, we must utilize the stub_status module alongside the Nginx Prometheus Exporter.

1. Configuring the Nginx Endpoint

First, ensure Nginx is installed and configured to serve the stub_status data. Modify the server configuration to enable the status module on a specific path.

server {
    listen 8080;
    server_name web-server.local;

    location /nginx_stats {
        stub_status;
        access_log off;
        allow 10.0.0.0/8;
        deny all;
    }

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}

After updating the configuration, restart the Nginx service to apply changes. Verify the status output is accessible by querying the endpoint.

2. Deploying the Metrics Exporter

The Nginx Prometheus Exporter acts as a bridge, reading from stub_status and exposing Prometheus-formatted metrics. We will deploy this as a sidecar or a standalone pod within the cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-stats-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-exporter
  template:
    metadata:
      labels:
        app: nginx-exporter
    spec:
      containers:
      - name: exporter
        image: nginx/nginx-prometheus-exporter:0.11.0
        args:
          - '-nginx.scrape-uri=http://nginx-service.default.svc.cluster.local:8080/nginx_stats'
        ports:
        - containerPort: 9113
          name: metrics-port

This Deployment launches the exporter container configured to scrape the Nginx statistics endpoint exposed in the previous step.

3. Exposing the Exporter Service

To make the metrics accessible to the Prometheus Operator, we need a Service object that selects the exporter pods.

apiVersion: v1
kind: Service
metadata:
  name: nginx-exporter-service
  labels:
    app: nginx-exporter
spec:
  ports:
  - port: 9113
    targetPort: 9113
    name: http
    protocol: TCP
  selector:
    app: nginx-exporter

4. Configuring Service Discovery

Finally, create a ServiceMonitor custom resource. This object defines how Prometheus should interact with the Service created above. It specifies the scrape interval and the port to target.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-monitoring
  namespace: monitoring
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: nginx-exporter
  namespaceSelector:
    matchNames:
      - default
  endpoints:
  - port: http
    interval: 30s
    path: /metrics