How does Apache Kafka achieve millions of messages per second throughput (Zero-Copy & Sequential I/O)?

Engineers often assume disks are inherently slow and memory is fast. Yet Apache Kafka sustains millions of writes/sec by exploiting fundamental OS kernel mechanisms and physical storage geometries.\n\n### 1. Sequential Disk I/O vs. Random Memory Access\n* Random access to RAM is fast (~100ns), but random access to hard disks/SSDs is slow (~1ms–10ms).\n* However, Sequential Disk Access bypasses seek overhead. A modern sequential disk write can achieve 600+ MB/s, which is often faster than random memory operations burdened by GC pauses and cache misses.\n* Kafka commits all incoming records as immutable append-only sequential commit logs.\n\n### 2. Linux OS Page Cache Utilization\nInstead of caching messages in JVM memory (which causes heap bloat, memory copy overhead, and devastating GC pauses), Kafka writes directly to the OS kernel's Page Cache:\n* All free physical server RAM is automatically utilized by Linux as file system disk cache.\n* If a producer sends a message and a consumer reads it moments later, the bytes are served straight from OS Page Cache without touching physical disk platters.\n* If Kafka is restarted, the OS Page Cache remains intact in kernel memory, eliminating cold-start latency.\n\n### 3. Zero-Copy Network Transfer (sendfile)\nIn traditional network servers, sending a file to a socket requires 4 context switches and 4 data copies:\n1. OS reads data from disk into OS cache.\n2. OS copies data into JVM application buffer.\n3. JVM copies data into OS socket buffer.\n4. OS copies socket buffer to Network Interface Card (NIC) buffer.\n\nKafka leverages the Linux kernel `sendfile()` system call (Zero-Copy):\n* Data is transferred directly from OS Page Cache straight to the NIC buffer via DMA (Direct Memory Access).\n* Completely eliminates CPU data copying into user space, allowing Kafka to saturate 100 Gbps network cards.\n\n### Key Architectural Comparison\n| Mechanism | Traditional Message Broker (RabbitMQ / ActiveMQ) | Apache Kafka Architecture |

| Consumer State | Server tracks individual message ACK per consumer | Consumer tracks its own 64-bit integer log offset |\n