Achieving End-to-End Exactly-Once Semantics in Apache Flink with Apache Kafka
Two-Phase Commit in Flink Sink Operators
Flink achieves exactly-once guarantees for external systems through an implementation of the Two-Phase Commit (2PC) protocol. The core mechanism aligns transaction boundaries with checkpoint barriers. When a sink operator receives the first record of a checkpoint interval or encounters a checkpoint barri ...
Posted on Sat, 13 Jun 2026 17:01:31 +0000 by mj99
Key Features and Enhancements in Apache Flink 1.14 to 1.17
Apache Flink 1.14.0 Highlights
Core Features
Checkpointing for Bounded Streams.
Mixed DataStream and Table/SQL Applications in Batch Execution Mode.
Introduction of the Hybrid Source for seamless reading across multiple sources.
Buffer Debloating to minimize checkpoint latency.
Fine-Grained Resource Management for dynamic Slot sizing.
New Puls ...
Posted on Thu, 07 May 2026 19:17:24 +0000 by kade119
Apache Flink Checkpoint Configuration Guide
Prerequisites
Exactly Once Processing
For exactly once semantics to work properly:
Source systems: Must support data retransmission (e.g., message queues like Kafka, distributed file systems like HDFS)
Sink systems: Must support idempotent operations (e.g., Doris supports deduplication)
At Least Once Processing
For at least once semantics:
S ...
Posted on Thu, 07 May 2026 06:14:41 +0000 by smpdawg