Environment Setup

• Operating System: CentOS 7
• Docker Version: 17.03.2-ce
• Docker Compose Version: 1.23.2

Docker Compose Configuration

To deploy Kafka with Zookeeper, create a docker-compose.yml file with the following content. This configuration avoids common issues like build failures and connection errors.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:2.11-0.11.0.3
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://:9092
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The volume mount /var/run/docker.sock enables containerized Docker commands to interact with the host system.

Starting the Services

Execute docker-compose up -d in the directory containing docker-compose.yml to download images and launch containers. Verify with docker ps:

CONTAINER ID   IMAGE                              COMMAND                  PORTS
abc123def456   wurstmeister/kafka:2.11-0.11.0.3   "start-kafka.sh"         0.0.0.0:32773->9092/tcp
ghi789jkl012   wurstmeister/zookeeper             "/bin/sh -c '/usr/..."   0.0.0.0:2181->2181/tcp

Containers are named kafka-docker_kafka_1 and kafka-docker_zookeeper_1.

Checking Versions

Run this command to display the Kafka version:

docker exec kafka-docker_kafka_1 find / -name '*kafka_*' | head -1 | sed 's/.*kafka_//'

Output example: 2.11-0.11.0.3, indicating Scala 2.11 and Kafka 0.11.0.3.

For Zoookeeper version:

docker exec kafka-docker_zookeeper_1 pwd

Output example: /opt/zookeeper-3.4.9, showing Zookeeper 3.4.9.

Scaling Kafka Brokers

Increase the number of brokers to four with:

docker-compose scale kafka=4

Confirm with docker ps to see four Kafka containers.

Creating a Topic

Define a topic named topic001 with four partitions and a replication factor of two:

docker exec kafka-docker_kafka_1 kafka-topics.sh --create --topic topic001 --partitions 4 --zookeeper zookeeper:2181 --replication-factor 2

List the topic from another container:

docker exec kafka-docker_kafka_3 kafka-topics.sh --list --zookeeper zookeeper:2181

Inspect topic details:

docker exec kafka-docker_kafka_3 kafka-topics.sh --describe --topic topic001 --zookeeper zookeeper:2181

Output displays partition leaders and replicas.

Consuming Messages

Start a consumer for topic001:

docker exec kafka-docker_kafka_2 kafka-console-consumer.sh --topic topic001 --bootstrap-server kafka-docker_kafka_1:9092,kafka-docker_kafka_2:9092,kafka-docker_kafka_3:9092,kafka-docker_kafka_4:9092

The console waits for encoming messages.

Producing Messages

In a new terminal, initiate a producer with the -it flag for interactive input:

docker exec -it kafka-docker_kafka_1 kafka-console-producer.sh --topic topic001 --broker-list kafka-docker_kafka_1:9092,kafka-docker_kafka_2:9092,kafka-docker_kafka_3:9092,kafka-docker_kafka_4:9092

Enter text messages; they will appear in the consumer console, confirming succesfull message flow.