Installation and Configuration

Kafka 3.x and latter versions include an embedded Zookeeper, simplifying the setup process. This guide demonstrates how to install and configure a single-node Kafka cluster.

Prerequisites

Ensure you have a Java Runtime Environment (JRE) installed on you're system.

Step 1: Download and Extract

Download the Kafka binary distribution from the official website and extract it.

tar -xzf kafka_2.13-3.7.1.tgz
cd kafka_2.13-3.7.1

Step 2: Configure Zookeeper

Since Kafka 3.x includes Zookeeper, you only need to configure it. Create a dedicated directory for Zookeeper data.

mkdir -p /opt/kafka/zookeeper-data
vi config/zookeeper.properties

Edit the zookeeper.properties file to specify the data directory:

# The directory where the snapshot is stored.
dataDir=/opt/kafka/zookeeper-data

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial synchronization phase can take
initLimit=10

# The number of ticks that can pass between sending a request and getting an acknowledgement
syncLimit=5

Step 3: Configure Kafka Broker

Configure the Kafka broker settings in the config/server.properties file.

vi config/server.properties

Example configuration:

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

# The port the server listens on for client connections.
listeners=PLAINTEXT://:9092

# A list of directories in which to store log files.
log.dirs=/opt/kafka/kafka-logs

# The default number of log partitions per topic.
num.partitions=1

# The number of threads handling network requests.
num.network.threads=3

# The number of threads handling disk I/O.
num.io.threads=8

# The number of messages to fetch in a single fetch request.
fetch.max.bytes=1048576

# The maximum size of a message that the server can receive.
socket.request.max.bytes=104857600

# The number of hours to keep a log file before deleting it.
log.retention.hours=168

Step 4: Start the Services

Start the Zookeeper and Kafka broker services in the background using the -daemon flag.

# Start Zookeeper
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

# Start Kafka Broker
bin/kafka-server-start.sh -daemon config/server.properties

Verify that both services are running by checking the process list.

jps
# Expected output should include QuorumPeerMain and Kafka

Command-Line Operations

Kafka provides several command-line tools for managing topics and interacting with the cluster.

Managing Topics with kafka-topics.sh

The kafka-topics.sh script is used to create, delete, list, and describe topics.

# Create a new topic named 'my-topic' with one partition and a replication factor of 1
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 --replication-factor 1

# List all available topics
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Describe the details of 'my-topic'
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-topic

Producing and Consuming Messages

You can use the console producer and consumer tools too send and receive messages.

Producing Messages

Start the console producer to send messages to the 'my-topic' topic.

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic
# Type messages and press Enter to send them. Press Ctrl+C to exit.

Consuming Messages

Start the console consumer to read messages from the 'my-topic' topic.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning
# Messages sent by the producer will appear here.