Handling Large Blobs & File Uploads at Scale
Handling Large Blobs & File Uploads at Scale
Streaming a 5 GB video file through your API Gateway consumes backend thread pools, eats memory buffers, and is vulnerable to complete failure if a cell tower disconnects at 99%.
1. Direct-to-Storage with Presigned URLs
Never pipe massive file payloads through your microservice containers!
- Client requests upload ticket:
POST /api/v1/uploads { filename: "vid.mp4", size: 5GB }. - Backend generates a cryptographically signed Amazon S3 / GCS Presigned URL with a 15-minute expiration.
- Client uploads directly to the S3 bucket using HTTP
PUT. - S3 fires an event notification (SNS / SQS) to trigger asynchronous transcoding pipelines.
2. Multipart & Resumable Uploads
- Splits the file into 5 MB–20 MB binary chunks on the client.
- Uploads chunks in parallel over multiple TCP streams.
- If Chunk #42 fails due to packet loss, only that 10 MB chunk is retried, not the entire 5 GB video.
Key Architecture Trade-Offs & Decision Matrix
| Decision Factor | Choice A | Choice B | Selected Architectural Rationale |
|---|---|---|---|
| Communication Protocol | REST over HTTP/2 | WebSockets / gRPC | Use REST for stateless CRUD APIs; use WebSockets for low-latency bidirectional streaming. |
| State Management | Centralized Distributed Store | Local Memory with Sticky Sessions | Centralized Redis / Database state allows seamless container auto-scaling and zero-downtime rolling deploys. |
| Concurrency Control | Optimistic Locking (version column) | Pessimistic Locking (SELECT FOR UPDATE) | Optimistic locking delivers superior throughput for low-conflict consumer operations. |
Related Topics
Expand your knowledge by learning about adjacent concepts in system design.
Consistent Hashing Architecture & Implementation
Deep dive into consistent hashing algorithms, virtual nodes, token rings, and minimal key remapping during cluster resizing in distributed databases.
Networking Essentials
Learn the important parts of networking that you'll need to know for your system design interviews
API Design
Master API Design for distributed systems and FAANG interviews: REST vs GraphQL vs gRPC, idempotency keys, cursor pagination, 202 async jobs, security, and RFC 7807 errors.
Cheat Sheet Utility
View and print a concise system design reference card.