Architecture Overview

Apache Kafka serves as a distributed event streaming platform capable of handling high-throughput, low-latency data pipelines. It funcsions as a commit log where data is organized into topics. Producers write records to these topics, while Consumers subscribe to them to process the data. To ensure horizontal scaling and fault tolerance, topics are split into partitions that are replicated across multiple nodes. This architecture makes Kafka ideal for real-time analytics, log aggregation, and event sourcing.

Installation and Setup

To deploy Kafka locally on a Windows environment, follow these steps to configure the necessary services and file structures.

1. Download and Extraction

Acquire the Kafka binary release (specifically version 3.6.1 compiled for Scala 2.12) which requires Java 8 or higher. Extract the archive to a directory with a short path to avoid potential file system length limits.

2. Configuration

Navigate to the config directory within your installation folder. You need to modify two key properties files to define storage locations.

• zookeeper.properties: Update the dataDir parameter to point to a dedicated folder for Zookeeper data storage.
• server.properties: Update the log.dirs parameter to specify the directory where Kafka will store its log segments.

3. Startup Scripts

For convenience, create batch files in the root installation directory to launch the services.

Create a file named start-zookeeper.bat containing the following command:

call bin/windows/zookeeper-server-start.bat config/zookeeper.properties

Similarly, create start-kafka.bat with the following content:

call bin/windows/kafka-server-start.bat config/server.properties

4. Service Management

The startup sequence is critical. You must launch Zookeeper before Kafka. Upon initialization, the data directories specified in the configuration files will be populated. When shutting down, terminate the Kafka process first, followed by Zookeeper to ensure clean state management.

Command Line Operations

Once the services are running, you can manage the cluster using the command-line tools located in the bin/windows directory.

Verifying Processes

Execute the jps command in a terminal. You should see the QuorumPeerMain (Zookeeper) and Kafka processes listed among the active Java processes.

Topic Management

Create a new topic named demo-stream with a single partition and replication factor:

bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --topic demo-stream --create --partitions 1 --replication-factor 1

List all available topics to verify creation:

bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --list

Inspect the metadata and partition details of the specific topic:

bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --topic demo-stream --describe

Producing and Consuming Data

To test the data flow, open two separate command windows.

In the first window, start a consumer to listen for messages:

bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic demo-stream --from-beginning

In the second window, start a producer to publish messages:

bin\windows\kafka-console-producer.bat --bootstrap-server localhost:9092 --topic demo-stream

Any text entered into the producer console will immediately appear in the consumer console, confirming the successful installation and operation of the pipeline.