Architectural Breakdown & AnalysisFAANG Aligned
Apache Kafka scales throughput by splitting topics into append-only commit log segments called Partitions.
1. Partition Routing & Ordering
- Messages with the same partition key (e.g.,
user_id) route to the exact same partition. - Kafka guarantees strict message ordering only within a single partition.
2. Consumer Group Scaling
- A Consumer Group is a cluster of workers co-operating to process topic messages.
- Each partition is assigned to exactly one consumer worker within a consumer group.
- Max Concurrency Limit: If a topic has 4 partitions and you launch 6 consumers in a group, 2 consumers will sit idle. To scale consumers, increase partition count.
3. Delivery Guarantees
- At-Most-Once: Commit offset before processing. Fast, but messages are lost if worker crashes.
- At-Least-Once: Commit offset after processing. Messages are re-delivered on failure; requires downstream handlers to be idempotent.
- Exactly-Once: Combines idempotent producers and transactional consumer offset commits.
TIP — Scaling Rule
Design topic partition counts based on expected peak consumer throughput requirements rather than initial broker node count.
Key Architectural Comparison
| Delivery Guarantee | Producer Config | Consumer Offset Commit | Risk of Duplicates | Performance Impact |
|---|---|---|---|---|
| At-Most-Once | acks=0 or acks=1 | Auto-commit offset *before* processing message. | Zero duplicates (Risk of data loss on crash). | Highest throughput. |
| At-Least-Once | acks=all, retries $> 0$ | Commit offset *after* message processing succeeds. | High risk of duplicate processing (requires idempotency). | Standard production baseline. |
| Exactly-Once (EOS) | enable.idempotence=true | Two-Phase Commit Transactional API across topics. | Zero duplicates, zero loss. | Moderate overhead (transactional state log). |