Dedicated Service User Creation

Creating a dedicated Kafka runtime user isolates the service from your primary system account to avoid permission conflicts and environment pollusion. This step is optional but strongly recommended for production and test environments.

# Create dedicated kafka service user, you will be prompted to set a password during execution
sudo adduser kafka-svc

# Grant sudo privileges to the new user for installation operations
sudo usermod -aG sudo kafka-svc

# Switch to the kafka-svc user session
su - kafka-svc

Java Runtime Configuration and Kafka Installation

Java Environment Setup

Kafka relies on Java 8 or higher runtime, run the following commands to install OpenJDK 8:

sudo apt update && sudo apt install -y openjdk-8-jdk
# Verify installation success
java -version
# Expected output sample:
# openjdk version "1.8.0_412"
# OpenJDK Runtime Environment (build 1.8.0_412-8u412-ga-1~22.04.1-b08)
# OpenJDK 64-Bit Server VM (build 25.412-b08, mixed mode)

Kafka Package Installation

Installing from source requires additional Gradle dependencies and may fail due to network issues, so precompiled binary packages are preferred:

# Create package storage directory
mkdir -p ~/pkg
# Download Kafka 3.7.1 precompiled binary for Scala 2.13
wget https://downloads.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz -P ~/pkg
# Extract to runtime directory
mkdir -p ~/kafka-runtime
tar -xzf ~/pkg/kafka_2.13-3.7.1.tgz --strip-components=1 -C ~/kafka-runtime

Core Configuration Adjustment

Kafka Broker Configuration

Edit the Kafka server configuration file to adapt to your network environment:

vim ~/kafka-runtime/config/server.properties

Modify the following core parameters:

# Bind to all network interfaces for external access
listeners=PLAINTEXT://0.0.0.0:9092
# Advertise your server's public IP to external clients, replace <YOUR_SERVER_IP> with actual address
advertised.listeners=PLAINTEXT://<YOUR_SERVER_IP>:9092
# Specify log storage path
log.dirs=/home/kafka-svc/kafka-runtime/kafka-logs

Zookeeper Configuration

Edit the embedded Zookeeper configuration file to adjust data storage path:

vim ~/kafka-runtime/config/zookeeper.properties

Modify the following parameters as needed:

# Replace default temporary directory with persistent storage path
dataDir=/home/kafka-svc/kafka-runtime/zk-data
# Client connection port, keep default 2181 if no conflict exists
clientPort=2181

Service Startup and Topic Management

Run the following commands to start services and create test topics:

# Start Zookeeper in daemon mode
~/kafka-runtime/bin/zookeeper-server-start.sh -daemon ~/kafka-runtime/config/zookeeper.properties
# Start Kafka broker in daemon mode
~/kafka-runtime/bin/kafka-server-start.sh -daemon ~/kafka-runtime/config/server.properties
# Create test topic with 2 partitions and 1 replica
~/kafka-runtime/bin/kafka-topics.sh --create --topic test-event-topic --partitions 2 --replication-factor 1 --bootstrap-server <YOUR_SERVER_IP>:9092
# View topic details
~/kafka-runtime/bin/kafka-topics.sh --describe --topic test-event-topic --bootstrap-server <YOUR_SERVER_IP>:9092
# Expected output sample:
# Topic: test-event-topic	TopicId: 7xQZ9t2mQYmXz8aB3cD9eA	PartitionCount: 2	ReplicationFactor: 1	Configs: 
# 	Topic: test-event-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
# 	Topic: test-event-topic	Partition: 1	Leader: 0	Replicas: 0	Isr: 0

Message Production and Consumpsion

Write Test Messages

Use the console producer to send test messages to the topic:

~/kafka-runtime/bin/kafka-console-producer.sh --topic test-event-topic --bootstrap-server <YOUR_SERVER_IP>:9092
# After entering the interactive mode, you can input messages line by line:
>user_login_event uid=10086
>order_paid_event order_id=20240601001
>system_alert_event level=warn
# Press Ctrl+C to exit the producer

Consume Stored Messages

Use the console consumer to read messages from the topic, add --from-beginning to read all historical messages:

~/kafka-runtime/bin/kafka-console-consumer.sh --topic test-event-topic --from-beginning --bootstrap-server <YOUR_SERVER_IP>:9092
# Expected output:
# user_login_event uid=10086
# order_paid_event order_id=20240601001
# system_alert_event level=warn

Common Troubleshooting

Silent Startup Failure in Daemon Mode

When starting services in daemon mode, no error will be printed directly even if startup fails. You can verify service status with the stop command:

~/kafka-runtime/bin/kafka-server-stop.sh
# If output shows "No kafka server to stop", the service failed to start

Common root causes:

1. Invalid IP configuration in advertised.listeners parameter
2. Insufficient write permission for the configured log or data directory
3. Port 9092 or 2181 is occupied by other processes, you can check with sudo netstat -tulpn | grep -E '9092|2181'