Design ChatGPT (LLM Streaming Inference Platform)

ChatGPT is a real-time conversational AI system serving hundreds of millions of users worldwide. Unlike traditional web services that compute a database query and return a static JSON payload, conversational AI platforms run multi-billion-parameter neural network weights across distributed GPU clusters, streaming generated tokens sequentially to clients with strict latency budgets.

1. System Requirements

Functional Requirements

1. Interactive Chat Prompts: Users can submit text prompts to ongoing or new chat conversations and receive AI-generated completions.
2. Real-time Token Streaming: The response must stream token-by-token back to the user as generated by the inference engine (Server-Sent Events / WebSockets).
3. Multi-Turn Context: Previous conversational turns within the session must be retained and incorporated into subsequent prompts up to the model's context limit.
4. Chat History Management: Users can list past conversations, retrieve full chat transcripts, rename threads, and delete history.
5. Generation Cancellation: Users can interrupt generation mid-stream, immediately halting GPU compute for that request.

Non-Functional Requirements

1. Time-To-First-Token (TTFT): Under 250ms for prompt processing and initial token emission.
2. Generation Latency (Inter-Token Latency): 30–50ms per token (20–30 tokens/second), matching or exceeding human reading speed.
3. Availability & Fault Tolerance: 99.99% uptime for the chat gateway; worker node crashes should not drop user connections.
4. Scalability: Support 200 Million Daily Active Users (DAU), 20,000 peak prompt QPS, and up to 150,000 concurrent streaming responses.
5. Cost & GPU Efficiency: Maximize GPU tensor core utilization via continuous batching, KV caching, and prompt caching.

2. Scale & Capacity Estimation

Traffic Calculations

• Daily Active Users (DAU): $200\text{ Million}$
• Average Prompts Per User: 6 prompts/day $\rightarrow 1.2\text{ Billion prompts/day}$
• Average QPS: $\frac{1.2 \times 10^9}{86,400} \approx 14,000\text{ QPS}$
• Peak QPS (2x multiplier): $\approx 28,000\text{ QPS}$
• Average Generation Time: 10 seconds per response (approx. 250 tokens @ 25 tokens/sec).
• Concurrent Active Streams: $28,000\text{ QPS} \times 10\text{s} = 280,000\text{ concurrent streaming connections}$.

Storage Estimation

• Average Prompt Length: 100 tokens (\approx 400 bytes).
• Average Completion Length: 250 tokens (\approx 1,000 bytes).
• Daily Conversation Data: $1.2 \times 10^9 \times 1.5\text{ KB} \approx 1.8\text{ TB/day}$.
• Annual Storage (with metadata & indexing): $\approx 800\text{ TB/year}$.

3. High-Level Architecture Blueprint

                                  +-----------------------+
                                  |   Client App / Web    |
                                  +-----------+-----------+
                                              |
                                              | HTTPS / WSS / SSE
                                              v
                                  +-----------------------+
                                  |    Cloudflare CDN     |
                                  |   (DDoS & TLS Term)   |
                                  +-----------+-----------+
                                              |
                                              v
                                  +-----------------------+
                                  |      API Gateway      |
                                  | (Auth, Rate Limiting) |
                                  +-----+-----------+-----+
                                        |           |
             +--------------------------+           +--------------------------+
             |                                                                 |
             v                                                                 v
+------------------------+                                         +------------------------+
|   Conversation Service |                                         |   Streaming Gateway    |
| (CRUD, History, Meta)  |                                         | (SSE Connection Hub)   |
+-----------+------------+                                         +-----------+------------+
            |                                                                  ^
            v                                                                  | Token Stream
+------------------------+         +-----------------------+                   |
|   PostgreSQL / Scylla  |         |     Kafka Message     |                   |
|  (Chat Messages DB)    |         |   Prompt Queue Pool   |                   |
+------------------------+         +-----------+-----------+                   |
                                               |                               |
                                               v                               |
                                   +-----------------------+                   |
                                   |    Inference Router   |                   |
                                   |   & KV Cache Arbiter  |                   |
                                   +-----------+-----------+                   |
                                               |                               |
                                               v Dispatch                      |
                                   +-----------------------+                   |
                                   |    GPU Worker Node    |                   |
                                   | (vLLM / PagedAttention|-------------------+
                                   | TensorRT-LLM Engine)  |
                                   +-----------------------+

4. Detailed Component Workflows

A. The Generation Request Lifecycle

1. Request Ingestion: User sends POST /api/v1/conversations/{id}/messages with the prompt text.
2. Context Assembly: The Conversation Service fetches the last $N$ turns from the conversation store. If the context exceeds token thresholds, historical turns are dynamically summarized.
3. Moderation & Safety Check: An asynchronous lightweight classifier filters out toxic prompts before invoking expensive GPU workloads.
4. Queue & Scheduling: The prompt, along with user priority tier (Free vs Plus/Pro), is enqueued into a distributed Kafka partition.
5. Inference Router Dispatch: The router checks which GPU worker node already has the KV cache loaded for that conversation_id (Prefix/Prompt Cache Affinity).
6. Continuous Batching Execution: The GPU worker adds the prompt to its running iteration batch. In each decoding step (forward pass):
7. New tokens are sampled from logits.
8. Emitted tokens are published to a Redis pub/sub channel or gRPC stream.
9. The Streaming Gateway writes the token to the client's active Server-Sent Event (SSE) connection.
10. Stream Finalization: An [DONE] sentinel token signals the end of generation. The complete completion is written back to the primary database.

5. Staff-Level Deep Dives

Deep Dive 1: KV Caching & PagedAttention

During transformer auto-regressive generation, computing attention requires the Key ($K$) and Value ($V$) matrices of all previous tokens in the sequence. Without caching, generation has $O(N^2)$ complexity.

• The Problem: Storing KV vectors in GPU HBM (High Bandwidth Memory) consumes huge capacity (e.g. 1-2 GB per long sequence). Traditional contiguous memory allocation causes up to 60-80% memory fragmentation.
• The Solution (PagedAttention): Inspired by OS virtual memory paging, PagedAttention partitions KV tensors into fixed-size blocks (e.g., 16 tokens per block).
• Blocks are stored in non-contiguous physical GPU memory spaces mapped by a page table.
• Enables dynamic memory allocation, allowing concurrent batch size on an 8x H100 node to increase by $2.5\times - 4\times$.

Deep Dive 2: Continuous Batching (Iteration-Level Scheduling)

• Static Batching Flaw: Traditional deep learning batches requests together. If Request A generates 20 tokens and Request B generates 500 tokens, the GPU sits idle waiting for Request B while Request A's slot is wasted.
• Continuous Batching: Operates at the iteration level rather than request level. Once Request A finishes its 20th token, it is immediately evicted from the batch, and a new prompt Request C is inserted into the batch for the next iteration without restarting the GPU pipeline.

Deep Dive 3: Client Connection Management & Resilient Streaming

• Server-Sent Events (SSE) vs WebSockets: SSE uses standard HTTP/2 multiplexing, automatic reconnects, and lightweight unidirectional streaming, eliminating the overhead of full-duplex WebSocket state tables.
• Disconnect Recovery: Every token chunk carries an incremental sequence number (seq_id). If a mobile client loses connection on cell towers, it reconnects using:

GET /api/v1/conversations/{id}/stream?last_seq_id=42

• The Streaming Gateway replays buffered tokens from an in-memory Redis ring buffer (TTL: 60s) before resuming live generation.

Deep Dive 4: Generation Cancellation & GPU Resource Reclaiming

• When a user clicks "Stop Generating", the client sends POST /api/v1/conversations/{id}/cancel.
• The API Gateway emits a cancellation signal carrying the request_id across the internal message bus.
• The GPU inference worker checks a local atomic cancellation flag before executing the next decoding step. If flagged, it evicts the sequence's KV page tables immediately, freeing memory back to the GPU pool in less than 50ms.

6. Database Schema & Data Models

Conversations Table (conversations)

CREATE TABLE conversations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL,
    title VARCHAR(255) NOT NULL,
    model_version VARCHAR(64) NOT NULL DEFAULT 'gpt-4o',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_archived BOOLEAN DEFAULT FALSE
);
CREATE INDEX idx_conversations_user ON conversations(user_id, updated_at DESC);

Messages Table (messages)

CREATE TABLE messages (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    conversation_id UUID NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
    sender_role VARCHAR(16) NOT NULL CHECK (sender_role IN ('system', 'user', 'assistant')),
    content TEXT NOT NULL,
    token_count INT NOT NULL,
    finish_reason VARCHAR(32) DEFAULT 'stop',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_messages_conversation ON messages(conversation_id, created_at ASC);

7. Trade-offs & Architecture Decisions