Ordered Pod Management and Stable Storage with Kubernetes StatefulSets

Kubernetes distinguishes between stateless and stateful workloads. Stateless components can be replaced at any time without data loss, whereas stateful systems—such as databases and distributed key-value stores—depend on node identity, local persistence, and startup order. While Custom Resource Definitions and Operators extend Kubernetes for domain-specific orchestration logic, the StatefulSet controller remains the native mechanism for menaging replicated stateful Pods.

Core Characteristics

StatefulSets are tailored for applications that require:

  • Stable network identifiers: Each Pod receives a predictable hostname based on the StatefulSet name and its ordinal index. Because Pod IPs are ephemeral, clients rely on DNS names that resolve directly to the Pod through a headless Service.
  • Dedicated persistent storage: Replicas cannot safely share a single backend volume because each instance typically holds distinct data. StatefulSets use volumeClaimTemplates to automatically provision a unique PersistentVolumeClaim (PVC) for each Pod.
  • Ordered lifecycle operations: Pod are created sequentially from index 0 to N-1 and terminated in reverse order from N-1 down to 0. This ordering ensures that dependencies—such as bootstrapping a primary node before secondaries—are respected.
  • Controlled rolling updates: Updates roll out in descending ordinal order, and administrators can partition the rollout to limit the blast radius.

Required Resources

A complete StatefulSet deployment relies on three cooperating objects:

  1. Headless Service (clusterIP: None): Creates DNS records for each Pod endpoint, enabling direct Pod-to-Pod discovery without load balancing.
  2. StatefulSet controller: Defines the Pod template, replica count, serviceName linking to the headless Service, and update strategy.
  3. volumeClaimTemplates: A template from which the controller generates PVCs. Each PVC binds to an independent PersistentVolume (PV), ensuring storage isolation across replicas.

Example Manifest

The following configuration deploys a three-replica StatefulSet fronted by a headless Service. Each replica requests its own 2 Gi volume.

apiVersion: v1
kind: Service
metadata:
  name: datastore
  labels:
    tier: backend
spec:
  ports:
  - port: 80
    name: http
  clusterIP: None
  selector:
    app: content-server
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: content-server
spec:
  serviceName: datastore
  replicas: 3
  selector:
    matchLabels:
      app: content-server
  template:
    metadata:
      labels:
        app: content-server
    spec:
      containers:
      - name: frontend
        image: nginx:1.25
        ports:
        - containerPort: 80
          name: http
        volumeMounts:
        - name: html
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: html
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 2Gi

Before applying the manifest, verify that sufficient PVs exist or that a dynamic provisioner is available. The controller automatically creates one PVC per Pod from the template.

Apply the manifest and inspect the resulting objects:

kubectl apply -f stateful-demo.yaml
kubectl get service,pvc,sts,pv

Because the Service is headless, each Pod is reachable at a deterministic DNS address:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

For example, the first replica resolves as content-server-0.datastore.default.svc.cluster.local.

Scaling and Deletion Behavior

Scaling respects ordinal ordering. Increasing the replica count appends Pods sequentially; decreasing it removes the highest-index Pod first.

kubectl scale statefulset content-server --replicas=5
kubectl patch statefulset content-server -p '{"spec":{"replicas":2}}'

Deleting the StatefulSet removes Pods in reverse order but leaves the PVCs intact. Recreating the object binds the new Pods to the existing claims, preserving data and identity across rescheduling.

kubectl delete -f stateful-demo.yaml
kubectl get pvc
kubectl apply -f stateful-demo.yaml
kubectl get pods

Rolling Updates and Partitioning

StatefulSets default to a RollingUpdate strategy that updates Pods from highest index to lowest. The partition field can restrict the update to a subset of replicas, which is useful for canary validation.

With five replicas (0 through 4) and partition set to 3, only Pods with an ordinal index greater than or equal to 3 receive the new image.

Configure a partition and change the container image:

kubectl patch statefulset content-server -p \
  '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":3}}}}'

kubectl set image statefulset/content-server frontend=nginx:1.26

After confirming that the updated replicas behave correctly, remove the partition to roll the change out to the remaining Pods:

kubectl patch statefulset content-server -p \
  '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'

Tags: kubernetes StatefulSet Persistent Storage Container Orchestration Headless Service

Posted on Sun, 01 Nov 2026 16:36:39 +0000 by DaveTheAve