Apache Flink Deep Dive (Stateful Stream Processing)

Apache Flink is the industry-standard framework for high-throughput, low-latency stateful stream processing, powering real-time fraud scoring, clickstream analytics, and continuous machine learning pipelines.

1. Stream Processing Fundamentals

Time Semantics

• Event Time: The timestamp when the event actually occurred on the client device (e.g. mobile sensor).
• Ingestion Time: The timestamp when the record entered the Kafka broker.
• Processing Time: The local machine clock time on the Flink task manager executing the operator.

Watermarks: Handling Out-of-Order Data

Because mobile networks cause packets to arrive delayed or out of order, Flink uses Watermarks: $$\text{Watermark}(t) = \max(\text{EventTime}) - \text{AllowedLateness}$$ A watermark of $t$ asserts: *"We assume no more events with timestamp $\le t$ will arrive."* When the watermark passes a window's end boundary, the window is finalized and evaluated.

2. Fault Tolerance: Chandy-Lamport Checkpointing & Exactly-Once

Flink achieves true Exactly-Once state guarantees without stopping the data stream:

1. Checkpoint Barriers: The JobManager periodically injects barrier tokens into the source streams.
2. Barriers flow downstream interleaved with normal data tuples.
3. When an operator receives barriers from all input channels, it snapshots its local state asynchronously to durable storage (S3/HDFS).
4. If any node crashes, all operators roll back their state to the last successful checkpoint and Kafka source offsets are reset.

3. State Backends: Memory vs. RocksDB

• HashMap State Backend: State lives on the JVM heap. Ultra-fast microsecond access, but limited by host RAM.
• EmbeddedRocksDBStateBackend: Out-of-core state engine. Data is stored in local NVMe disk backed by LSM-trees. Supports terabytes of state per task slot.

Key Architecture Trade-Offs & Decision Matrix