Event-Driven Architecture & Scaling with Apache Kafka

Apache Kafka is an append-only distributed event log capable of processing millions of events per second with high throughput, zero data loss, and sub-millisecond latencies.

1. Core Architecture: Topics, Partitions & Brokers

                                TOPIC: user-clicks
    +---------------------------------------------------------------+
    | Partition 0: [Offset 0] -> [Offset 1] -> [Offset 2] -> [Offset 3]
    +---------------------------------------------------------------+
    | Partition 1: [Offset 0] -> [Offset 1] -> [Offset 2] -> [Offset 3]
    +---------------------------------------------------------------+
    | Partition 2: [Offset 0] -> [Offset 1] -> [Offset 2] -> [Offset 3]
    +---------------------------------------------------------------+

• Partition as the Unit of Scalability: An individual partition cannot be split across brokers. Ordering is strictly guaranteed only within a single partition, not across partitions.
• Partition Key Routing: Producers compute hash(key) % N_partitions (MurmurHash2). All messages for a given user_id land on the exact same partition, guaranteeing chronological ordering.

2. Consumer Groups & Dynamic Rebalancing

• Consumers belong to a Consumer Group. Each partition in a topic is consumed by exactly one consumer within that group.
• If a consumer dies, the Group Coordinator triggers a Rebalance: unassigned partitions are reassigned to surviving consumers, with offsets resumed from the internal __consumer_offsets topic.

3. Durability & Replication Mechanics (acks=all)

• In-Sync Replicas (ISR): The set of replica nodes that have fully caught up with the partition leader.
• Producer Acknowledgments:
• acks=0: Fire-and-forget (fastest, high risk of loss).
• acks=1: Acknowledged once the partition leader writes to disk.
• acks=all (-1): Acknowledged only after all in-sync replicas write the message. Guarantees zero data loss even if the leader node suffers catastrophic hardware failure.

Key Architecture Trade-Offs & Decision Matrix