Design ChatGPT (LLM Inference Platform)

ChatGPT is an AI inference platform serving large language model responses with low-latency token streaming.

1. High-Level Design

Large Language Models generate text sequentially (token-by-token). Due to high GPU compute cost and size, responses must be streamed back as they generate.

Client App <--- SSE/WebSockets <--- Inference Node (Worker) <--- Prompt Queue (Kafka) <--- API

Components

1. Inference Gateway: Receives user prompts, attaches context history, and queues request.

2. Prompt Dispatcher: Prioritizes prompts and dispatches to appropriate GPU worker nodes.

3. Inference Worker: Hosts the LLM weights in GPU memory. Runs tensor parallel calculations.

4. Streaming Broker: Establishes Server-Sent Events (SSE) channels to stream generated tokens to the client app in real time.

2. Potential Deep Dives

• KV Caching Optimization:

During LLM generation, previous token keys and values are cached in GPU memory to avoid redundant matrix operations. Platforms partition KV cache spaces dynamically (like virtual memory paging) to optimize memory utilization.

• Dynamic Batching:

Combines multiple single prompt evaluations arriving within milliseconds into a single batched tensor execution step on the GPU.

3. References & Tech Blogs

• OpenAI Blog: Scaling LLM Inference Infrastructure
• vLLM: PagedAttention for High-Throughput KV Caching