Kafka, a distributed publish-subscribe messaging system, is designed for high-throughput handling of real-time data feeds. Its architecture offers O(1) time complexity for data persistence, enabling efficient access even with terabytes of data. Capable of processing over 100,000 messages per second on commodity hardware, Kafka supports partitioned topics and distributed consumption, ensuring message order within partitions. It facilitates both batch and real-time data processing and scales horizontally with ease.

Core Kafka Concepts

• Broker: A Kafka server instance within a cluster.
• Topic: A category or feed name to which messages are published.
• Partition: A topic is divided into one or more partitions, which are physical storage units.
• Producer: An application that publishes messages to Kafka brokers.
• Consumer: A client application that subscribes to topics and reads messages.
• Consumer Group: A logical grouping of consumers that collectively consume from a topic. Each partition is assigned to a single consumer within a group at any given time.

Kafka APIs

Kafka provides four primary APIs:

1. Producer API: For publishing messages to topics.
2. Consumer API: For subscribing to topics and processing messages.
3. Streams API: For building stream processing applications that consume from and produce to topics.
4. Connector API: For creating reusable producers or consumers to integrate Kafka with external data systems.

Common Kafka Interview Questions and Concepts

• Kafka Use Cases: Kafka is employed for system decoupling, asynchronous communication, and buffering high-volume traffic (peak shaving). It serves as a messaging system, a durable storage system due to its disk persistence and replication, and a foundation for stream processing platforms.

• AR, ISR, and Leader Election: AR (Assigned Replicas) refers to all replicas of a partition. ISR (In-Sync Replicas) are those replicas that are actively synchronized with the leader. The leader tracks the ISR. If a follower falls too far behind or becomes unresponsive, it's removed from the ISR. Conversely, a follower that catches up can be added back. By default, only replicas in the ISR are eligible to become the new leader during a leader failure. replica.lag.time.max.ms (default 10s) defines the maximum lag for a follower to remain in the ISR. unclean.leader.election.enable (default true) allows a non-ISR replica to become leader, increasing availability at the risk of data loss.

• Watermarks (HW, LEO, LSO, LW):

  • HW (High Watermark): The offset up to which consumers can read. It's determined by the minimum LEO among ISR replicas.
  • LEO (Log End Offset): The offset of the next message to be written in a partition log. Each replica maintains its LEO.
  • LSO (Log Start Offset): The starting offset of a log file. It can be modified by operations like deleteRecords.
  • LW (Low Watermark): The minimum logStartOffset across all replicas in the AR set.

• Message Ordering: Message order is guaranteed only within a single partition. By using a partitioner that consistently routes messages with the same key to the same partition, order for specific keys can be preserved.

• Producer Interceptors, Serializers, and Partitioners: The processing order for a producer is Interceptor -> Serializer -> Partitioner. Interceptors can modify messages before they are sent or prepare for callbacks. Serializers convert objects to byte arrays. Partitioners determine the target partition for a message if one isn't explicitly specified.

• Producer Client Architecture: The producer client uses two threads: the main thread for creating messages and caching them in the RecordAccumulator, and a Sender thread that efficiently batches and sends these messages to brokers.

• Consumer Offset Management: Historically, older consumer clients stored offsets in ZooKeeper. Modern clients store offsets in an internal Kafka topic, __consumer_offsets. When committing offsets, offset + 1 is typically stored to indicaet the next message to be consumed.

• Consumer Group Scalability: If the number of consumers in a group exceeds the number of partitions for a subscribed topic, some consumers will not receive any messages. Custom partition assignment strategies can be implemented to redistribute partitions if needed.

• Duplicate and Missed Messages: Duplicate messages can occur due to producer retries, consumer rebalances before offsets are committed, or client restarts before auto-committed offsets are finalized. Missed messages can happen with auto-commit if a consumer fails after committing an offset but before processing the corresponding message, or if producers use acks=0 and data is lost before replication.

• Multi-threaded Consumption: To achieve multi-threaded consumption, each thread should have its own KafkaConsumer instance (thread confinement). A common pattern is to use a dedicated thread for fetching messages and a thread pool for processing them, decoupling fetching from processing.

• Topic Partitioning and Replication: When a topic is created or altered, Kafka creates directories for partitions and associated metadata in ZooKeeper. The number of partitions can be increased using kafka-topics.sh --alter. Decreasing partitions is not directly supported due to complexities with message ordering and data handling. When choosing partition counts, consider throughput requirements and broker availability; too many partitions can increase leader election times during outages. Replication is managed via replication.factor, ensuring data redundancy. The min.insync.replicas setting is crucial for data integrity, requiring a minimum number of replicas to acknowledge writes.

• Kafka Configuration Parameters: Key broker configurations include broker.id, zookeeper.connect, log.dirs, message.max.bytes, num.network.threads, num.io.threads, log.segment.bytes, log.retention.hours, log.retention.bytes, and delete.topic.enable. Producer configurations like bootstrap.servers, acks, buffer.memory, compression.type, retries, batch.size, and linger.ms are vital. Consumer configurations such as group.id, auto.offset.reset, enable.auto.commit, fetch.min.bytes, and partition.assignment.strategy control consumption behavior.

• Producer acks Mechanism: acks=0 offers no guarantee. acks=1 ensures the leader has written the message. acks=all guarantees the leader and all in-sync replicas have written the message, providing the strongest durability.

• Kafka Segments: Partition data is stored in segment files (.log for data, .index for index). Segment files are named by the offset of their first message. This structure facilitates efficient log cleanup and message retrieval.

Kafka Installation and Tooling

• Installation: Kafka installation typically involves downloading the distribution, setting environment variables, and starting ZooKeeper and Kafka server processes.

• Visualization Tools: Several GUI tools aid in managing Kafka clusters:

  • Kafka-Eagle: Provides a web-based interface for monitoring and managing Kafka clusters, requiring configuration for ZooKeeper, Kafka, and MySQL.
  • Kafka-Manager: A Yahoo-developed tool for cluster management, accessible via a web UI after compilation and configuration of ZooKeeper hosts.
  • Offset Explorer (formerly Kafka Tool): A cross-platform GUI application for browsing topics, messages, and consumer offsets, offering features for developers and administrators.

Production Environment Troubleshooting

• Offset Expiration: In older Kafka versions, offsets.retention.minutes could lead to offset data loss if topics weren't consumed for extended periods, causing duplicate consumption. Increasing this retention period or manually resetting offsets (kafka-consumer-groups.sh --reset-offsets) are solutions.

• Adding Kafka Nodes: To rebalance data across newly added brokers, use the kafka-reassign-partitions.sh script. This involves defining topics to move, generating a reassignment plan, and executing it.

• Cross-Cluster Synchronization: MirrorMaker is used for replicating data between Kafka clusters, essential for disaster recovery or data migration. It acts as a consumer on the source cluster and a producer on the target cluster.

• Network Traversal (Gateways): Enabling Kafka access across different network zones (e.g., internal vs. external) requires careful configuration of listeners and advertised.listeners in server.properties, alongside proper IP/domain mapping in host files and network gateway configurations.

• Data Loss and Duplicates: Data loss typically stems from producer configurations (acks) or network issues. Duplicate messages often result from auto-commit misconfigurations or producer retries without idempotency. Implementing manual offset commits and ensuring at-least-once or exactly-once processing semantics addresses these.

• Message Backlog (Backpressure): High message production rates exceeding consumption capacity can lead to backlogs. Solutions include increasing topic partitions, optimizing consumer processing logic (e.g., asynchronous processing), scaling consumer instances, or addressing unedrlying Kafka cluster performance issues.