Prerequisites

• Kubernetes version: 1.23.17
• Zookeeper version: 3.5.9
• Kafka version: 3.6.0

Zookeeper Cluster Setup

Create a StatefulSet with three pods, each running a Zookeeper server instance:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk-cluster
spec:
  serviceName: zk-headless
  replicas: 3
  selector:
    matchLabels:
      app: zk-node
  template:
    metadata:
      labels:
        app: zk-node
    spec:
      containers:
      - name: zk-container
        image: zookeeper:3.5.9
        ports:
        - containerPort: 2181
        - containerPort: 2888
        - containerPort: 3888
        env:
        - name: ZOO_MY_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        command:
        - sh
        - -c
        - |
          echo ${HOSTNAME##*-} > /data/myid
          exec zkServer.sh start-foreground
        volumeMounts:
        - name: zk-data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: zk-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

Verifying Zookeeper Cluster

Check server IDs and status:

for pod in $(kubectl get pods -l app=zk-node -o name); do
  kubectl exec $pod -- cat /data/myid
  kubectl exec $pod -- zkServer.sh status
done

Kafka Cluster Deployment

Deploy Kafka brokers with Zookeeper coordination:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-cluster
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka-node
  template:
    metadata:
      labels:
        app: kafka-node
    spec:
      containers:
      - name: kafka-broker
        image: confluentinc/cp-kafka:6.2.0
        env:
        - name: KAFKA_BROKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zk-0.zk-headless:2181,zk-1.zk-headless:2181,zk-2.zk-headless:2181"
        ports:
        - containerPort: 9092
        volumeMounts:
        - name: kafka-data
          mountPath: /var/lib/kafka
  volumeClaimTemplates:
  - metadata:
      name: kafka-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

Testing Kafka Operations

Create a test topic:

kubectl exec kafka-0 -- \
  kafka-topics.sh --create \
  --bootstrap-server kafka-headless:9092 \
  --replication-factor 2 \
  --partitions 3 \
  --topic test-topic

Produce and consume messages:

# Producer
kubectl exec kafka-0 -- \
  kafka-console-producer.sh \
  --broker-list kafka-headless:9092 \
  --topic test-topic

# Consumer
kubectl exec kafka-1 -- \
  kafka-console-consumer.sh \
  --bootstrap-server kafka-headless:9092 \
  --topic test-topic \
  --from-beginning