API Gateway

An API Gateway is a mission-critical edge component in distributed systems and microservices architectures. Positioned as the single entry point (reverse proxy and ingress controller) between external client applications (mobile devices, web single-page apps, third-party partners, IoT fleets) and internal backend services, it decouples the external client contract from internal microservice topologies.

Rather than allowing clients to invoke dozens of private microservices directly—which causes severe network chattiness, security vulnerabilities, and tight coupling—the API Gateway centralizes cross-cutting concerns: dynamic traffic routing, zero-trust authentication & token exchange, distributed rate limiting, circuit breaking & fault tolerance, protocol mediation (REST to gRPC), and Backend-for-Frontend (BFF) aggregation.

1. Architectural Role & The Modern Edge Tier

In modern production architectures, the edge tier is layered. A frequent mistake in system design interviews is conflating Hardware Load Balancers (L4), Reverse Proxies (L7), API Gateways (L7 Application), and Service Meshes (L7 East-West).

The Edge Tier Decision Matrix

2. End-to-End Request Lifecycle & Edge Pipeline

When an external client dispatches an HTTP/JSON request to your platform, the API Gateway processes it through a strict, asynchronous execution pipeline before forwarding packets upstream:

[ Client: Mobile / Web ]
           │  HTTPS / HTTP/2 or HTTP/3 (QUIC)
           ▼
┌────────────────────────────────────────────────────────────────────────┐
│                        API GATEWAY EDGE PIPELINE                       │
│                                                                        │
│  1. Edge Ingress: TLS/SSL Termination (ALPN, Session Resumption)       │
│  2. DDoS & IP Reputation: WAF verification & Geo-IP blocking           │
│  3. Edge Authentication: Verify JWT Signature against cached JWKS      │
│  4. Context Normalization: Inject W3C traceparent & X-Identity-Passport│
│  5. Distributed Rate Limiter: Redis Token Bucket check (429 if empty)  │
│  6. Route Matcher: Path/Host resolution & Dynamic Service Discovery    │
│  7. Protocol Transcoder: JSON/HTTP payload -> gRPC/Protobuf binary     │
│  8. Resiliency Wrapper: Circuit Breaker, Timeout Budget & Pool Guard   │
│  9. Upstream Forwarding: Send request over warmed HTTP/2 / gRPC pool   │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │  mTLS (gRPC / HTTP/2)
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                        INTERNAL MICROSERVICES                          │
│                                                                        │
│  [ Order Service ]       [ User Service ]       [ Inventory Service ]  │
└────────────────────────────────────────────────────────────────────────┘

Detailed Pipeline Execution Stages

1. Edge Ingress & TLS Termination: The gateway terminates client TLS/SSL handshakes at the edge. By utilizing ALPN (Application-Layer Protocol Negotiation), it negotiates HTTP/2 or HTTP/3 (QUIC) with clients, converting multiplexed client streams into persistent, pipelined internal TCP connections.
2. DDoS Mitigation & WAF Filtering: Checks against IP blacklists, bot signatures, SQL injection patterns, and Cross-Site Scripting (XSS) vectors before touching application layers.
3. Authentication & Token Validation: Inspects the Authorization: Bearer <JWT> header, cryptographically verifies the signature against cached public keys, and rejects expired or tampered requests immediately.
4. Header Sanitization & Context Injection: Strips sensitive external headers (e.g., untrusted client X-Forwarded-For or forged internal user headers) and injects distributed tracing spans (traceparent, tracestate) and user identity claims.
5. Distributed Rate Limiting: Evaluates the client's quota bucket in Redis. If depleted, it halts execution and returns HTTP 429 Too Many Requests with a Retry-After header.
6. Routing & Dynamic Discovery: Queries the internal routing tree to map /api/v1/orders/* to the appropriate upstream cluster using weighted round-robin or least-request balancing.
7. Protocol Mediation & Transformation: Transcodes JSON payloads into gRPC binary formats if the target service communicates over Protobuf.
8. Resiliency & Circuit Breaking: Validates that the target cluster circuit breaker is CLOSED. If OPEN, it serves a graceful fallback or cached response without overloading the failing service.
9. Upstream Execution & Egress Pipeline: Dispatches the call, streams the response back, compresses the output (Brotli or Gzip), logs access telemetry, and records Prometheus latency metrics.

3. Traffic Routing, Dynamic Discovery & Envoy xDS Architecture

A primary duty of the API Gateway is dynamic routing: resolving incoming URIs, HTTP methods, and headers to active, healthy microservice IP addresses.

Routing Mechanisms in Production

• Path-Based Routing:
• /api/v1/checkout/* → checkout-cluster:8080
• /api/v1/catalog/* → catalog-cluster:8081
• Host / Subdomain-Based Routing:
• api.company.com → Core REST Gateway
• stream.company.com → WebSocket Streaming Gateway
• partners.company.com → B2B Gateway with strict mTLS
• Header & Metadata-Based Routing (Canary / Blue-Green Deployments):
• Inspecting X-User-Cohort: beta or X-Canary-Release: true to divert a percentage of real production traffic to a newly deployed canary service version.
• Weighted Traffic Splitting:
• Splitting incoming traffic 95% to v1.4.0 and 5% to v1.5.0 without requiring DNS propagation delays.

The Dynamic Discovery Problem & Envoy xDS

Traditional reverse proxies like standard NGINX required rewriting configuration files on disk and invoking nginx -s reload whenever container IP addresses changed. In dynamic cloud environments (Kubernetes, AWS ECS) where pods scale up, crash, and restart continuously, static reloads cause socket churn, connection dropouts, and CPU spikes.

Modern production gateways (e.g., Envoy Proxy) solve this with dynamic xDS APIs over streaming gRPC connections:

• LDS (Listener Discovery Service): Dynamically configures listening ports, TLS certificates, and IP bindings.
• RDS (Route Discovery Service): Dynamically updates HTTP route matching rules and virtual hosts.
• CDS (Cluster Discovery Service): Discovers upstream service clusters and their operational parameters (timeouts, health check intervals).
• EDS (Endpoint Discovery Service): Dynamically streams active upstream IP addresses and ports directly from Kubernetes EndpointSlices or Consul without restarting the gateway process.

Production Envoy Route Configuration Example

static_resources:
  listeners:
  - name: ingress_edge_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 443 }
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain: { filename: "/etc/certs/fullchain.pem" }
              private_key: { filename: "/etc/certs/privkey.pem" }
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: edge_routes
            virtual_hosts:
            - name: api_service
              domains: ["api.beingsde.in"]
              routes:
              - match: { prefix: "/api/v1/orders" }
                route:
                  cluster: order_service_cluster
                  timeout: 2.5s
                  retry_policy:
                    retry_on: "5xx,connect-failure,reset"
                    num_retries: 2
                    per_try_timeout: 800ms
              - match: { prefix: "/api/v1/users" }
                route:
                  cluster: user_service_cluster
                  timeout: 1.5s
          http_filters:
          - name: envoy.filters.http.jwt_authn
          - name: envoy.filters.http.ratelimit
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

4. Edge Security: Zero-Trust "Passport" Exchange & Authentication

A critical architectural pitfall in microservices is forcing every individual downstream service to parse, validate, and verify asymmetric cryptographic signatures of public OAuth2 JSON Web Tokens (JWTs).

The "Identity Passport" Token Exchange Pattern

• The Problem: Public JWTs signed with RSA-256 or ECDSA require non-trivial CPU cycles for cryptographic verification. If a single user request fans out to 10 microservices, verifying that identical public token 10 times wastes massive CPU resources and requires every service to connect to an external Identity Provider (Auth0, Okta, Keycloak) to fetch JWKS keys.
• The Solution: The API Gateway performs Token Exchange:
• The client sends a standard asymmetric JWT in Authorization: Bearer <token>.
• The API Gateway validates the signature against its in-memory cached JWKS public key set.
• The Gateway extracts claims (User ID, Tenant ID, Scopes, Roles).
• The Gateway strips the heavy external Bearer token.
• The Gateway mints an internal, compact, cryptographically signed "Identity Passport" header (or attaches authenticated headers like X-User-Id: 9821, X-Tenant-Id: acme-corp, X-User-Roles: admin,billing).
• Downstream services accept these headers over mutual TLS (mTLS) or verify a fast symmetric HMAC-SHA256 signature, eliminating redundant external cryptographic overhead.

Client (Mobile App)
      │
      │  HTTPS Request with External RS256 JWT
      ▼
┌─────────────────────────────────────────────────────────────┐
│                     API GATEWAY EDGE                        │
│                                                             │
│  1. Verify RS256 signature with cached JWKS public keys     │
│  2. Validate 'exp', 'nbf', 'iss', and 'aud' claims          │
│  3. Strip untrusted client Authorization header             │
│  4. Mint internal signed Identity Passport                  │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               │  mTLS with internal X-Identity-Passport
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                 DOWNSTREAM MICROSERVICES                    │
│                                                             │
│  Accept internal identity context without querying Auth0    │
└─────────────────────────────────────────────────────────────┘

5. Distributed Rate Limiting & Traffic Shaping at Scale

Rate limiting protects upstream services from cascading outages, noisy neighbors, brute-force credential stuffing, and DDoS floods. In distributed systems, naive in-memory counters fail because client requests are balanced across dozens of stateless gateway instances.

Rate Limiting Algorithm Comparison

Production Token Bucket Redis Lua Script

To prevent race conditions between reading token balances and deducting quotas across concurrent gateway threads, rate limiting must execute atomically in Redis using a Lua script:

-- Keys: [1] rate_limit_key
-- Args: [1] max_tokens (capacity), [2] refill_rate (tokens/sec), [3] requested_tokens, [4] current_timestamp_sec

local key = KEYS[1]
local max_capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])
local now = tonumber(ARGV[4])

-- Retrieve current state: tokens and last_refreshed_timestamp
local data = redis.call("HMGET", key, "tokens", "last_updated")
local current_tokens = tonumber(data[1])
local last_updated = tonumber(data[2])

if current_tokens == nil then
    -- First time initializing the bucket
    current_tokens = max_capacity
    last_updated = now
else
    -- Calculate tokens accumulated since last request
    local elapsed = math.max(0, now - last_updated)
    local tokens_to_add = elapsed * refill_rate
    current_tokens = math.min(max_capacity, current_tokens + tokens_to_add)
    last_updated = now
end

-- Check if sufficient tokens exist
if current_tokens >= requested then
    current_tokens = current_tokens - requested
    redis.call("HMSET", key, "tokens", current_tokens, "last_updated", last_updated)
    redis.call("EXPIRE", key, math.ceil(max_capacity / refill_rate) * 2)
    return { 1, math.floor(current_tokens) } -- Allowed: 1, Remaining tokens
else
    redis.call("HMSET", key, "tokens", current_tokens, "last_updated", last_updated)
    return { 0, math.floor(current_tokens) } -- Rejected: 0, Remaining tokens
end

Standardized Rate Limit Response Headers

When a request is evaluated, the API Gateway returns standardized HTTP headers adhering to IETF draft specifications:

HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 12
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1726261200

{
  "type": "https://api.beingsde.in/errors/rate-limit-exceeded",
  "title": "Too Many Requests",
  "status": 429,
  "detail": "Exceeded quota of 100 requests per minute. Bucket refills in 12 seconds.",
  "instance": "/api/v1/orders/submit"
}

6. Resilience Engineering: Circuit Breaking, Timeouts & Bulkheading

When an upstream microservice experiences high latency or memory saturation, client retries and incoming requests will quickly exhaust gateway worker threads and connection pools, leading to a total system outage.

The 3 Circuit Breaker States

                 [ Normal Operation ]
                   ┌──────────────┐
                   │    CLOSED    │
                   └──────┬───────┘
                          │ Failure threshold exceeded (e.g. > 50% errors)
                          ▼
                   ┌──────────────┐
                   │     OPEN     │ ◄─── Fast-fail all requests immediately
                   └──────┬───────┘      (Return 503 / cached fallback)
                          │
                          │ Cooldown sleep timer expires (e.g. 5 seconds)
                          ▼
                   ┌──────────────┐
                   │  HALF-OPEN   │
                   └──────┬───────┘
            Success rate  │  Failure occurs
            exceeds 90%   │
     ┌────────────────────┴────────────────────┐
     ▼                                         ▼
┌──────────────┐                          ┌──────────────┐
│    CLOSED    │                          │     OPEN     │
└──────────────┘                          └──────────────┘

• CLOSED State: Requests flow directly upstream. The gateway monitors failure rates and p99 latency across rolling windows.
• OPEN State: When errors exceed the failure threshold (e.g., 50% failures over 10 seconds), the circuit trips to OPEN. The gateway immediately fast-fails incoming requests with HTTP 503 Service Unavailable or a cached fallback, shielding the struggling service from traffic.
• HALF-OPEN State: After a configured sleep window (e.g., 5,000ms), the gateway allows a canary probe (e.g., 5% of requests) through. If successful, the circuit resets to CLOSED; if failures persist, it immediately reverts to OPEN.

Upstream Timeout Budgets & Deadline Propagation

Without deadline propagation, an upstream service that takes 5 seconds to respond will cause upstream gateways and downstream microservices to block indefinitely.

• The gateway establishes an absolute request timeout budget (e.g., 2,000ms).
• The gateway propagates this budget downstream using standard headers: X-Request-Deadline: 1726261200500 or gRPC grpc-timeout: 1800m.
• If a downstream database query detects that the client's timeout budget has already expired, it aborts immediately rather than wasting CPU cycles on a dead request.

The Danger of Blind Retries & Full Jitter Backoff

Blindly retrying failed requests during an outage amplifies traffic by 3x or 5x, creating a catastrophic Retry Storm.

7. Backend-for-Frontend (BFF) & Request Aggregation

Modern platforms serve radically different client surfaces: mobile iOS/Android apps over cellular connections, desktop web single-page applications over fiber broadband, and smart watch devices with micro-displays.

The Over-Fetching & Under-Fetching Dilemma

• A mobile app rendering a home feed would need to invoke 5 separate microservices:
• GET /api/v1/users/profile (User Service)
• GET /api/v1/orders/recent (Order Service)
• GET /api/v1/notifications/unread (Notification Service)
• GET /api/v1/recommendations (ML Recommender)
• GET /api/v1/loyalty/points (Rewards Service)
• Over high-latency 4G/5G mobile networks, establishing 5 sequential HTTP requests causes severe battery drain and 2,000ms+ page load times.

The BFF Pattern

The gateway hosts a specialized Backend-for-Frontend (BFF) aggregation endpoint (GET /api/v1/mobile/home-dashboard):

// Conceptual BFF Orchestration inside API Gateway
export async function getMobileHomeDashboard(userId: string, authPassport: string) {
  // Execute independent microservice calls concurrently over internal low-latency network
  const [profileRes, ordersRes, notificationsRes, recommendationsRes] = await Promise.allSettled([
    userService.getProfile(userId, { headers: { 'X-Passport': authPassport } }),
    orderService.getRecentOrders(userId, { headers: { 'X-Passport': authPassport } }),
    notificationService.getUnread(userId, { headers: { 'X-Passport': authPassport } }),
    recommendationService.getFeed(userId, { timeout: 400 }) // Strict 400ms SLA
  ]);

  // Graceful degradation: If recommendations fail or time out, return default cached list
  return {
    user: profileRes.status === 'fulfilled' ? profileRes.value : null,
    recentOrders: ordersRes.status === 'fulfilled' ? ordersRes.value.slice(0, 3) : [],
    unreadNotifications: notificationsRes.status === 'fulfilled' ? notificationsRes.value.count : 0,
    recommendations: recommendationsRes.status === 'fulfilled' 
      ? recommendationsRes.value 
      : fallbackRecommendationsCache.get(userId)
  };
}

8. Protocol Mediation: REST to gRPC Transcoding

While external mobile and web clients communicate predominantly via standard HTTP/JSON and REST, internal microservices achieve orders-of-magnitude higher throughput, lower serialization latency, and strict type safety using gRPC and Protocol Buffers (Protobuf).

How gRPC-JSON Transcoding Works

1. Service teams define their API contracts once using .proto files with HTTP annotations:

   syntax = "proto3";
   package ecommerce.v1;
   import "google/api/annotations.proto";

   service OrderService {
     rpc GetOrder(GetOrderRequest) returns (OrderResponse) {
       option (google.api.http) = {
         get: "/api/v1/orders/{order_id}"
       };
     }
   }

   message GetOrderRequest {
     string order_id = 1;
   }

1. The Protobuf compiler generates an Envoy Binary Descriptor Set (`order_service.pb`).
2. Envoy's native C++ grpc_json_transcoder filter dynamically parses incoming HTTP GET requests, extracts {order_id} from the URI path, serializes it directly into a binary GetOrderRequest buffer, and dispatches the RPC call over an existing multiplexed HTTP/2 channel to the backend service.
3. The binary OrderResponse is transcoded back to JSON with zero custom gateway code.

9. Real-World Architecture: Event Loops vs. Thread-per-Request

When evaluating gateway architectures in interviews, understanding the underlying I/O concurrency model is paramount.

Netflix Zuul 1 vs. Zuul 2 / Envoy Proxy

10. The 45-Minute FAANG Interview Blueprint

When tasked with "Design an API Gateway" in a FAANG system design interview, follow this structured 5-phase breakdown:

Phase 1: Clarifying Requirements & Scale (5 Minutes)

• Functional Scope:
• Request routing, authentication offloading, rate limiting, and metrics collection.
• Protocol transcoding (HTTP to gRPC) and response aggregation (BFF).
• Non-Functional Requirements:
• Ultra-Low Latency: Gateway overhead must add < 5ms at p99.
• High Availability: 99.999% uptime across multiple geographic availability zones.
• Scale: Support 500,000 QPS peak traffic with graceful degradation under spikes.

Phase 2: High-Level Topology Diagram (10 Minutes)

• Draw the complete North-South request flow:
• DNS / Geo-Anycast routing to nearest AWS Route 53 / Cloudflare POP.
• Layer 4 Network Load Balancer (AWS NLB) terminating TCP.
• Auto-scaling API Gateway cluster (Envoy instances in Kubernetes).
• Redis Cluster for centralized rate limiting and token bucket counters.
• Downstream internal microservices communicating over mTLS.

Phase 3: Core Architectural Deep Dives (18 Minutes)

• Walk the interviewer through your Distributed Rate Limiter: Explain why you chose the Token Bucket algorithm, show how the Redis Lua script guarantees atomicity, and explain fallback behavior if Redis is partitioned (fail-open vs fail-closed).
• Detail the Identity Passport Pattern: Show how offloading asymmetric JWT validation at the edge saves downstream service CPU cycles and simplifies microservice development.
• Explain your Circuit Breaker Configuration: Define exact thresholds (e.g., trip to OPEN if 50% of calls fail over a 10s window; sleep 5s before canary probe in HALF-OPEN).

Phase 4: Edge Cases & Operational Resiliency (8 Minutes)

• What happens if Redis dies? Fall back to local in-memory sliding window counters on each gateway pod with a temporary 20% quota relaxation to prevent dropping legitimate user traffic.
• How do you handle clock skew across gateway nodes? Rely on Redis server time (TIME command) rather than individual host clocks for sliding window calculations.
• How do you prevent a slow microservice from crashing the gateway? Apply connection pool bulkheading with dedicated maximum connection queues per cluster.

Phase 5: Interview Summary Checklist (4 Minutes)

• Confirm that external clients never communicate directly with internal IPs.
• Reiterate that sensitive client headers are stripped at the perimeter.
• Verify that all asynchronous downstream calls adhere to strict deadline propagation.