Apache Flink (Distributed Stream Processing)

Apache Flink is a stateful distributed stream processing framework designed for low-latency calculations over real-time event streams.

1. Core Architecture

Unlike batch processing tools (like Hadoop or traditional Spark MapReduce) that partition data in bulk chunks, Flink processes records individually or in micro-second stream windows.

Data Source (Kafka) ---> Flink JobManager (Orchestrator) ---> TaskManager (Calculates State) ---> Sink (Redis/DB)

Key Abstractions

• Streams & Transformations: Operations (map, flatMap, filter, keyBy) transform inputs into logical streams.
• Stateful Compute: Flink maintains local, in-memory states (ValueState, ListState) backed by RocksDB for fault tolerance.
• Watermarks: Monotonically increasing timestamps injected into streams to manage late or out-of-order events.
• Windows: Groups events by time (Tumbling, Sliding, Session) or count.

2. State & Exactly-Once Semantics

• Checkpointing (Chandy-Lamport Algorithm):

Flink regularly writes snapshots of TaskManager states to distributed storage (S3/HDFS) asynchronously.

• Exactly-Once End-to-End Processing:

Achieved by combining Kafka idempotent transactional writers, Flink checkpoint offsets, and Two-Phase Commit Sinks. If a failure occurs, the engine rolls back task registers and resumes logs from the last valid checkpoint.

3. References & Tech Blogs

• Apache Flink Documentation: Stateful Stream Processing
• Netflix Tech Blog: In-Memory Stream Processing with Flink