Environment Setup

JDK Installation

yum install -y java-1.8.0

ZooKeeper Cluster Configuration

Downloading ZooKeeper

wget https://archive.apache.org/dist/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
tar xf apache-zookeeper-3.7.0-bin.tar.gz

Configurasion File Adjustments

# Backup sample configuration
# cp zoo_sample.cfg zoo.cfg

cd conf/
vim zoo.cfg

Configuration parameters:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/server-data/zookeeper
clientPort=2181
server.1=10.10.101.32:2888:3888
server.2=10.10.101.33:2888:3888
server.3=10.10.101.34:2888:3888

Creating myid Files

On each node:

# Node 1
[root@test-k8s-master1 zookeeper]# cat myid 
1

# Node 2
[root@test-k8s-master2 zookeeper]# cat myid 
2

# Node 3
[root@test-k8s-master3 zookeeper]# cat myid 
3

Systemd Service Registration

Create service file:

vim /lib/systemd/system/zookeeper.service

Service definition:

[Unit]
Description=zookeeper.service
After=network.target
ConditionPathExists=/data/server-side/zookeeper/conf/zoo.cfg

[Service]
Type=forking
User=root
Group=root
ExecStart=/data/server-side/zookeeper/bin/zkServer.sh start
ExecStop=/data/server-side/zookeeper/bin/zkServer.sh stop

[Install]
WantedBy=multi-user.target

Kafka Cluster Deployment

Kafka Download

wget https://mirrors.bfsu.edu.cn/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz

Server Properties Configuration

vim config/server.properties

Key settings:

broker.id=0
zookeeper.connect=10.10.101.32:2181,10.10.101.33:2181,10.10.101.34:2181

Systemd Integration

vim /lib/systemd/system/kafka.service

Service configuration:

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target
After=network.target zookeeper.service

[Service]
Type=simple
WorkingDirectory=/data/server-side/kafka
ExecStart=/data/server-side/kafka/bin/kafka-server-start.sh config/server.properties
ExecStop=/data/server-side/kafka/bin/kafka-server-stop.sh config/server.properties
Restart=on-failure

[Install]
WantedBy=multi-user.target

Kafka Manager Installation

Kafka Manager provides a web-based interface for managing Kafka clusters

Download and Extract

kafka-manager-2.0.0.0.zip
unzip kafka-manager-2.0.0.0.zip

Configuration Setup

vim conf/application.conf

Update connection settings:

kafka-manager.zkhosts=[zk_ip1:zk_port1,zk_ip2:zk_port2,zk_ip3:zk_port3]
http.port=9000

Enable JMX Monitoring

Modify kafka-server-start.sh to enable JMX port:

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="8999"
fi

Launch Menager

nohup ./kafka-manager -Dconfig.file=/data/server-side/kafka-manager/conf/application.conf >/dev/null 2>&1 &

Access via browser:

[IP:9000]