Apache ZooKeeper Deep Dive (Consensus & Coordination)

ZooKeeper provides centralized coordination services for large distributed systems (Kafka, Hadoop, HBase), managing leader elections, service discovery registries, and distributed locks.

1. Data Model & ZNode Types

ZooKeeper structures data like an in-memory hierarchical Unix filesystem:

• Persistent ZNodes: Remain stored until explicitly deleted.
• Ephemeral ZNodes: Automatically deleted by the cluster when the client session disconnects or heartbeats time out. Perfect for leader election and service presence.
• Sequential ZNodes: Append a monotonically increasing 10-digit number to the path (e.g. /lock/node-0000000001). Used for FIFO distributed locking.

2. The ZAB Protocol (ZooKeeper Atomic Broadcast)

ZooKeeper runs the proprietary ZAB consensus protocol:

• Leader Election Phase: Nodes discover each other and elect the candidate with the highest epoch and transaction ID ($zxid$).
• Atomic Broadcast Phase: All state updates flow through the leader, which proposes transactions to follower nodes via two-phase commit over TCP.
• Quorum Rule: Writes succeed once a simple majority of nodes ($Q = \lfloor N/2 \rfloor + 1$) acknowledge the write to disk.

Key Architecture Trade-Offs & Decision Matrix