Overview of Apache Zookeeper

Apache Zookeeper is a high-performance coordination service for distributted applications. It exposes a simple set of primitives—often used to implement higher-level services for synchronization, configuration maintenance, and groups/naming. It is designed to be highly available and reliable, forming the backbone of many big data ecosystems.

Core Characteristics of Zookeeper

• Leader-Follower Model: A Zookeeper cluster consists of a single Leader and multiple Followers. The Leader handles write requests, while Followers handle read requests and participate in the voting process.
• Quorum-based Reliability: The cluster remains operational as long as a majority (quorum) of nodes are active. This is why Zookeeper clusters are typically deployed with an odd number of nodes (e.g., 3, 5, or 7).
• Strict Consistency: Every client sees the same view of the service regardless of the server it connects to.
• Atomicity: Updates either succeed or fail completely; there are no partial results.
• Sequential Consistency: Updates from a specific client are applied in the order they were sent.

Zookeeper Data Model

Zookeeper's data is organized in a hierarchical namespace, much like a standard file system. Each node in this hierarchy is called a ZNode. Unlike files in a traditional system, ZNodes can store data (default limit is 1MB) and have children simultaneously. ZNodes can be persistent or ephemeral (deleted when the session ends).

The Zookeeper Leader Election Mechanism

Zookeeper uses the ZAB (Zookeeper Atomic Broadcast) protocol to maintain consistency. Leader election is a critical phase of this protocol.

Initial Election Logic

1. Self-Vote: When a node starts, it enters the LOOKING state and casts a vote for itself.
2. Vote Exchange: Nodes broadcast their vote (containing Server ID and Transaction ID).
3. Comparison: If a node receives a vote from another node with a higher ID (myid), it updates its own vote to match the superior candidate.
4. Quorum Reached: Once a node receives a majority of votes, it is promoted to LEADING status, and others transition to FOLLOWING.

Runtime Election Criteria

If the Leader fails, a new election is triggered based on the following priority sequence:

1. Epoch: The node with the highest logic clock (Epoch) wins.
2. ZXID: If Epochs are equal, the node with the most recent transaction ID (ZXID) wins, as it has the most up-to-date data.
3. SID: If both Epoch and ZXID are equal, the node with the highest Server ID (SID) wins.

Deploying a Zookeeper Cluster

Prepare three nodes with Java 8+ installed. Assume IP addresses: 192.168.10.11, 192.168.10.12, and 192.168.10.13.

Configuration (zoo.cfg)

# Basic Heartbeat and Timeout settings
tickTime=2000
initLimit=10
syncLimit=5

# Data Storage
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/logs
clientPort=2181

# Cluster Nodes (Format: server.id=IP:QuorumPort:ElectionPort)
server.1=192.168.10.11:2888:3888
server.2=192.168.10.12:2888:3888
server.3=192.168.10.13:2888:3888

Initializing Node Identity

On each server, create a myid file in the dataDir containing the corresponding server ID.

# On Server 1
echo "1" > /var/lib/zookeeper/data/myid
# On Server 2
echo "2" > /var/lib/zookeeper/data/myid
# On Server 3
echo "3" > /var/lib/zookeeper/data/myid

Understanding Apache Kafka

Kafka is a distributed event streaming platform capable of handling trillions of events a day. It is used for building real-time data pipelines and streaming apps.

Why Use Kafka?

• Decoupling: Producers and consumers operate independently.
• Peak Load Buffering: Kafka acts as a buffer to prevent downstream systems from being overwhelmed during traffic spikes.
• Asynchronous Communication: Allows background processing of data without blocking the main application flow.
• Fault Tolerance: Data is replicated across multiple brokers to prevent loss.

Architectrue Terminology

• Broker: A single Kafka server. A group of brokers forms a cluster.
• Topic: A logical category or feed name to which records are published.
• Partition: Topics are divided into partitions for parallelism. Each partition is an ordered, immutable sequence of records.
• Producer: The entity that sends data to Kafka brokers.
• Consumer: The entity that reads data from Kafka brokers.
• Offset: A unique identifier to each record within a partition, used by consumers to track their progress.

Kafka Cluster Implementation

Kafka relies on Zookeeper for cluster metadata management and controller election.

Server Configuration (server.properties)

Key parameters to modify on each broker:

# Unique ID for each broker
broker.id=1 

# Network listeners
listeners=PLAINTEXT://192.168.10.11:9092

# Data storage path
log.dirs=/var/local/kafka/data

# Retention policy (7 days)
log.retention.hours=168

# Zookeeper connection string
zookeeper.connect=192.168.10.11:2181,192.168.10.12:2181,192.168.10.13:2181

Managing Kafka via CLI

Once the cluster is running, use the following commands to manage topics and data:

1. Create a Topic:

kafka-topics.sh --create --bootstrap-server 192.168.10.11:9092 \
--replication-factor 2 --partitions 3 --topic app-logs

2. List Topics:

kafka-topics.sh --list --bootstrap-server 192.168.10.11:9092

3. Produce Messages:

kafka-console-producer.sh --broker-list 192.168.10.11:9092 --topic app-logs

4. Consume Messages:

kafka-console-consumer.sh --bootstrap-server 192.168.10.11:9092 \
--topic app-logs --from-beginning

5. Modify Partition Count:

kafka-topics.sh --bootstrap-server 192.168.10.11:9092 --alter \
--topic app-logs --partitions 5