Graceful Degradation & Healthiness Probes
The 3-Line Engineering Rule
Do not let a slow non-critical dependency fail your Kubernetes liveness probe; degrade gracefully and shed non-essential load.
If your `/health/liveness` check verifies database connectivity and the database encounters a 5-second lock, Kubernetes kills all application pods simultaneously, causing a full cascading outage.
Liveness probe: verifies only if process thread is alive (internal deadlock check). Readiness probe: verifies ability to accept traffic. Degrade non-critical features (recommendations, analytics) to keep core flows working.
Architectural Blueprint & Sequence
Hover over nodes to inspect state transitionsKubernetes Kubelet
Probes every 10s
Liveness Probe
Checks process deadlock ONLY
Readiness Probe
Checks DB/Kafka connection
App Service Pod
Serving live traffic
Degradation Fallback
Serve cached or static default
Key Engineering Takeaways
- Liveness probes should NEVER check downstream databases, caches, or third-party APIs.
- Readiness probes determine if the pod should receive traffic from the service load balancer.
- Implement graceful degradation: if recommendations service fails, return static fallbacks.
- Support Load Shedding: reject low-priority traffic with HTTP 503 before CPU saturates.
The Problem & Production Reality
Kubernetes liveness and readiness probes are frequently misunderstood. A liveness probe tells Kubernetes when to kill and restart a container. If you check an external database inside your liveness probe and the database slows down, Kubernetes restarts all 50 pods at once, compounding the database connection storm and prolonging the outage.
Anti-Pattern: Database Check in Liveness Probe
Configuring the liveness probe to verify database connectivity. When the database is overloaded, Kubernetes reboots all pods simultaneously.
// ❌ ANTI-PATTERN: Fatal Kubernetes Misconfiguration
// deployment.yaml
livenessProbe:
httpGet:
path: /actuator/health # ❌ Verifies DB, Redis, RabbitMQ!
port: 8080
# When DB hits connection limits, K8s kills all pods, guaranteeing total downtime!Production Pattern: Separated Liveness vs Readiness & Fallbacks
Configure Spring Boot Actuator's dedicated `/health/liveness` and `/health/readiness` endpoints with explicit probe groupings.
// ✅ PRODUCTION PATTERN: Separated K8s Probes
// application.yml
management:
endpoint:
health:
probes:
enabled: true
health:
livenessstate:
enabled: true # Checks only internal JVM health
readinessstate:
enabled: true # Verifies DB, Redis connection pool
// deployment.yaml
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 20
periodSeconds: 5
// Graceful Degradation in code:
public List<ProductDto> getRecommendations(String userId) {
try {
return recommendationClient.getPersonalized(userId);
} catch (Exception ex) {
log.warn("Recommendation service degraded. Serving fallback trending products.");
return staticFallbackCatalog.getTopTrending(); // Core checkout remains functional!
}
}Trade-off Analysis & Staff Recommendations
| Aspect | Pros / Strengths | Cons / Hazards | Staff Recommendation |
|---|---|---|---|
| Restarting Pod vs Removing from Ingress | Readiness failure stops new traffic without incurring the CPU cost of a JVM cold restart. | If a pod never recovers readiness, it sits idle until manually terminated. | Use readiness for dependencies; use liveness strictly for internal thread death. |
| Load Shedding | Dropping 10% of non-critical traffic allows the service to sustain the remaining 90%. | Requires identifying non-critical endpoints and returning HTTP 503 early. | Implement a servlet filter checking CPU or concurrency limits to reject traffic before OOM. |