beingsde
Sign In
Being SDE — Coding Design & Architectural Best Practices

Production Coding Design & 25 Best Practice Articles

Battle-tested engineering rules, fail-safe retries, idempotency keys, state machines, and concurrency locking patterns curated for Staff & Principal SDEs.

25 Engineering BlueprintsInteractive Scribble SchematicsProduction Code Snippets
Showing 25 of 25 practices
Resilience & RetriesCRITICAL
7 min read
RULE:

Never retry failed message queue consumers in a tight, synchronous loop on the main topic.

WHY:

A single corrupted message (poison pill) or transient dependency failure halts the entire consumer partition, introducing head-of-line blocking and catastrophic lag.

FIX:

Use a dedicated Retry Topic with exponential backoff + jitter, bounded maximum attempts (e.g. 3), and route to a Dead Letter Queue (DLQ) with PagerDuty alerts.

CONSUMER RETRY PIPELINESCRIBBLE BLUEPRINT
Non-blocking retry topic topology with exponential backoff and automated DLQ routing
#1

Event Producer

Publishes orders

#2

Main Topic

orders.events

#3

Consumer Worker

Throws 503 / DB timeout

#4
Exponential Delay

Retry Topic (Delay)

orders.retry-10s (Jitter)

#5
Max 3 Retries

Dead Letter Queue

orders.dlq + Alerts

#6

Manual / Batch Replay

Admin fix & re-ingest

Execution Sequence & Transitions
Event ProducerMain Topic[publish]
Main TopicConsumer Worker[poll]
Consumer WorkerRetry Topic (Delay)[fail: attempt <= 3]
Retry Topic (Delay)Consumer Worker[re-attempt after backoff]
Consumer WorkerDead Letter Queue[fail: attempt > 3]
Dead Letter QueueManual / Batch Replay[diagnose & replay]
#Kafka#RabbitMQ#DLQ#Exponential Backoff#Poison Pill
Includes Code & Trade-off MatrixRead Article
Resilience & RetriesCRITICAL
8 min read
RULE:

Never retry blind POST requests without idempotency keys, and always protect external calls with a circuit breaker and backoff jitter.

WHY:

Blind retries on payment or mutation APIs cause double charges and thundering herds that take down recovering third-party services.

FIX:

Combine Circuit Breaker (fail-fast on elevated error rates) + Exponential Backoff with Full Jitter + strictly enforce Idempotency-Key headers on mutating POST/PUT requests.

CIRCUIT BREAKER & RETRY STATESCRIBBLE BLUEPRINT
Circuit Breaker state transitions integrated with jittered idempotency retries
#1

Your Application

POST /v1/charges

#2
50% failure trips to OPEN

Circuit Breaker

CLOSED (Healthy)

#3

Retry Engine

Jitter + IdempotencyKey

#4

Partner Gateway

Returns 503 Service Unavailable

#5

Degraded Fallback

Queue for async reconciliation

Execution Sequence & Transitions
Your ApplicationCircuit Breaker[execute call]
Circuit BreakerPartner Gateway[state = CLOSED]
Partner GatewayRetry Engine[503 timeout / transient err]
Retry EnginePartner Gateway[backoff + jitter (attempt 2/3)]
Circuit BreakerDegraded Fallback[state = OPEN (fail-fast)]
#Resilience4j#Circuit Breaker#Exponential Backoff#Full Jitter#Idempotency
Includes Code & Trade-off MatrixRead Article
Observability & LoggingCRITICAL
7 min read
RULE:

Every log entry must carry a globally unique `trace_id` and `span_id` propagated across all microservices, async threads, and message queues.

WHY:

Without trace IDs, debugging a production error across 6 microservices and 3 queues requires cross-referencing timestamps, turning a 2-minute root cause into a 4-hour wild goose chase.

FIX:

Inject `trace_id` and `span_id` at the API Gateway via W3C `traceparent` headers, preserve them in Slf4j MDC, and forward them in Kafka/RabbitMQ headers.

DISTRIBUTED CONTEXT PROPAGATIONSCRIBBLE BLUEPRINT
Trace context flowing through HTTP ingress, async thread pools, and messaging queues
#1

Client Request

POST /orders

#2
Inject traceparent

API Gateway

Generate trace_id: 8f9b2c

#3

Order Service

MDC: trace=8f9b2c span=01

#4

Kafka Broker

Header: traceparent=8f9b2c

#5

Billing Worker

MDC: trace=8f9b2c span=02

#6

Log Aggregator

Search {trace_id='8f9b2c'}

Execution Sequence & Transitions
Client RequestAPI Gateway[HTTP Request]
API GatewayOrder Service[W3C traceparent header]
Order ServiceKafka Broker[produce with headers]
Kafka BrokerBilling Worker[consume & extract trace]
Order ServiceLog Aggregator[JSON log output]
Billing WorkerLog Aggregator[JSON log output]
#OpenTelemetry#MDC#trace_id#span_id#B3 Propagation
Includes Code & Trade-off MatrixRead Article
Observability & LoggingESSENTIAL
6 min read
RULE:

Do not log inside tight loops or echo full request/response bodies; log only state transitions, business milestones, and actionable errors.

WHY:

Excessive logging wastes millions in cloud ingestion costs (Datadog/Splunk), degrades disk I/O, drowns critical alerts in noise, and leaks customer PII.

FIX:

Use structured JSON with strict log levels (DEBUG for dev, INFO for key lifecycle milestones, WARN for handled anomalies, ERROR for action-required failures). Never log PII.

LOG VOLUME HYGIENESCRIBBLE BLUEPRINT
Replacing noisy loop spew with high-yield structured milestone telemetry
#1
$10k/mo Datadog bill

❌ NOISY: 10,000 logs/sec

log.info('Processing item: ' + id)

#2
Structured & Actionable

✅ INSIGHTFUL: 1 log

log.info('Batch complete', total=10000, failed=2, durationMs=340)

#3

PII Sanitizer

Mask email, phone, card tokens

#4

ERROR = Actionable

If no on-call action needed -> WARN

Execution Sequence & Transitions
❌ NOISY: 10,000 logs/sec✅ INSIGHTFUL: 1 log[refactor to batch metrics]
✅ INSIGHTFUL: 1 logPII Sanitizer[scrub sensitive fields]
✅ INSIGHTFUL: 1 logERROR = Actionable[strict log level triage]
#Log Hygiene#Structured JSON#Log Aggregation#Sampling#PII Masking
Includes Code & Trade-off MatrixRead Article
API Contracts & ErrorsCRITICAL
8 min read
RULE:

Any mutating API (POST/PUT) that performs money transfers, order creation, or external side-effects must be strictly idempotent.

WHY:

Network timeouts, user double-clicks, and automated retries will duplicate financial transactions or duplicate physical shipments without idempotency guarantees.

FIX:

Accept an `Idempotency-Key` header, acquire an atomic Redis lock (`SETNX` with TTL), check for existing cached responses, and return the identical result without re-executing logic.

IDEMPOTENCY EXECUTION FLOWSCRIBBLE BLUEPRINT
Atomic Redis state checking, duplicate lock detection, and cached response replaying
#1

Client App

POST /v1/payments (Key: abc-123)

#2

Redis State Lookup

GET idempotency:abc-123

#3
Lock Active

Is PROCESSING?

Return 409 Conflict / Wait

#4
Zero Side-Effects

Is COMPLETED?

Return Cached HTTP 200

#5

Execute Payment

Charge card & write DB

#6

Cache Response

Save payload + TTL: 24h

Execution Sequence & Transitions
Client AppRedis State Lookup[Header: Idempotency-Key]
Redis State LookupIs PROCESSING?[status == PROCESSING]
Redis State LookupIs COMPLETED?[status == COMPLETED]
Redis State LookupExecute Payment[key NOT found (acquire lock)]
Execute PaymentCache Response[success]
Cache ResponseClient App[HTTP 201 Created]
#Idempotency-Key#Redis SETNX#Distributed Lock#Double Submit#Stripe Pattern
Includes Code & Trade-off MatrixRead Article
Database & PerformanceCRITICAL
7 min read
RULE:

Cache only data with high read-to-write ratios and tolerable eventual consistency; never use Redis as a band-aid for missing database indexes.

WHY:

Over-caching dynamic data introduces stale reads, race conditions during cache invalidation, cache stampedes (thundering herd), and memory leaks.

FIX:

Use the Cache-Aside pattern with mandatory TTL + random jitter. Never cache volatile data updated on every user interaction without strict invalidation guarantees.

CACHE-ASIDE PATTERN & JITTERSCRIBBLE BLUEPRINT
Read flow through Redis with DB fallback, TTL jitter, and write invalidation
#1

Application Service

GET /products/492

#2

Redis Cache

Cache Key: prod:492

#3

Cache HIT (0.8ms)

Return JSON payload directly

#4

Cache MISS

Read from DB replica

#5

PostgreSQL DB

SELECT * FROM products WHERE id=492

#6

Populate Cache

SET prod:492 EX (3600s + Jitter)

Execution Sequence & Transitions
Application ServiceRedis Cache[1. Check Cache]
Redis CacheCache HIT (0.8ms)[found]
Redis CacheCache MISS[not found]
Cache MISSPostgreSQL DB[2. Query DB]
PostgreSQL DBPopulate Cache[3. Populate Redis with TTL]
Populate CacheApplication Service[4. Return result]
#Cache-Aside#Cache Stampede#TTL Jitter#Eviction Policies#Cache Invalidation
Includes Code & Trade-off MatrixRead Article
Data & State ConsistencyCRITICAL
8 min read
RULE:

Model multi-step business entities (inventory, orders, payments) as strict Finite State Machines with explicit transition tables.

WHY:

Free-form status columns updated with arbitrary strings lead to impossible states (e.g. shipping an item that was cancelled or refunding a returned item twice).

FIX:

Enforce transition guards, immutable state history logs, and reject illegal state transitions at both domain code and database constraints.

DETERMINISTIC STATE TRANSITIONSSCRIBBLE BLUEPRINT
Finite State Machine graph showing valid transitions, guard conditions, and terminal nodes
#1

AVAILABLE

In warehouse stock

#2

RESERVED

Checkout started (TTL: 15m)

#3

ALLOCATED

Payment succeeded

#4
Terminal State

DISPATCHED

Handed to courier (TERMINAL)

#5

RELEASED

Cart expired / cancelled

Execution Sequence & Transitions
AVAILABLERESERVED[reserve()]
RESERVEDALLOCATED[payment_success]
ALLOCATEDDISPATCHED[ship_order]
RESERVEDRELEASED[timeout / cancel]
RELEASEDAVAILABLE[re-stock]
#Finite State Machine#Event-Driven#Audit Trail#Inventory#Terminal State
Includes Code & Trade-off MatrixRead Article
Data & State ConsistencyCRITICAL
9 min read
RULE:

Never perform a dual-write (e.g. write to Database AND publish to Kafka) in a single method without the Transactional Outbox pattern.

WHY:

Dual-writes have no atomic guarantee; if the DB commit succeeds but Kafka drops connection, downstream systems fall permanently out of sync.

FIX:

Persist the domain entity AND the outbound event into the same relational database in one ACID transaction, then use Change Data Capture (Debezium) or a reliable poller to publish.

TRANSACTIONAL OUTBOX PATTERNSCRIBBLE BLUEPRINT
Atomic DB transaction writing both entity and outbox table to eliminate dual-write hazards
#1

Client Request

POST /orders

#2
One DB Transaction

Atomic ACID Tx

BEGIN TRANSACTION

#3

orders Table

INSERT order (ID: 101)

#4

outbox Table

INSERT event payload

#5

Debezium CDC / Poller

Reads outbox table via WAL

#6

Kafka Event Broker

Publish OrderCreated event

Execution Sequence & Transitions
Client RequestAtomic ACID Tx[create order]
Atomic ACID Txorders Table[write state]
Atomic ACID Txoutbox Table[write event]
outbox TableDebezium CDC / Poller[tail WAL / query]
Debezium CDC / PollerKafka Event Broker[at-least-once publish]
#ACID#Transactional Outbox#Saga Pattern#Dual Writes#Eventual Consistency
Includes Code & Trade-off MatrixRead Article
Database & PerformanceCRITICAL
8 min read
RULE:

Never check stock with a simple `SELECT` and decrement with a separate `UPDATE`; use atomic database updates or distributed Lua scripts.

WHY:

During flash sales, 1,000 concurrent requests reading remaining stock = 1 will all believe stock is available, leading to massive overselling.

FIX:

Use atomic SQL `UPDATE stock = stock - 1 WHERE id = :id AND stock >= 1` or Redis atomic Lua scripts for flash-sale throughput.

CONCURRENCY LOCKING STRATEGIESSCRIBBLE BLUEPRINT
Comparing naive race conditions against Atomic SQL and In-Memory Redis Lua decr
#1
Race Condition

❌ Check-Then-Act Race

Read qty=1 -> Wait -> Write qty=0 (Oversold!)

#2
ACID Guaranteed

✅ Atomic SQL Conditional

UPDATE items SET qty=qty-1 WHERE qty>=1

#3
50,000+ QPS Flash Sale

⚡ Redis Atomic Lua

if redis.call('get', key) >= qty then decr

Execution Sequence & Transitions
❌ Check-Then-Act Race✅ Atomic SQL Conditional[migrate to atomic check]
✅ Atomic SQL Conditional⚡ Redis Atomic Lua[scale for flash sales]
#Optimistic Locking#Pessimistic Locking#SELECT FOR UPDATE#Redis Lua#Flash Sale
Includes Code & Trade-off MatrixRead Article
API Contracts & ErrorsESSENTIAL
7 min read
RULE:

Validate all untrusted input at the network perimeter; fail fast with descriptive 400 Bad Request envelopes before touching domain logic.

WHY:

Missing input validation allows SQL injection, payload-induced denial of service (ReDoS), corrupted databases, and security exploits.

FIX:

Enforce strict schema validation (Jakarta Bean Validation / Zod), sanitize strings, bound list sizes, and reject unexpected JSON fields.

BOUNDARY VALIDATION SHIELDSCRIBBLE BLUEPRINT
Fail-fast validation barrier rejecting malformed payloads before touching database layers
#1

External Client

POST /v1/users

#2
Fail Fast Boundary

Perimeter Validator

Schema, length, regex checks

#3
Zero DB load

400 Bad Request

RFC 7807 field validation errors

#4

Domain Logic

Safe sanitized business entities

#5

Clean Database

Valid, consistent data records

Execution Sequence & Transitions
External ClientPerimeter Validator[Raw JSON]
Perimeter Validator400 Bad Request[Invalid field (e.g. qty < 1)]
Perimeter ValidatorDomain Logic[Valid & Sanitized]
Domain LogicClean Database[Persist]
#Bean Validation#Fail Fast#Zod#Boundary Limits#SQL Injection
Includes Code & Trade-off MatrixRead Article
Resilience & RetriesCRITICAL
7 min read
RULE:

Never use default/infinite timeouts on HTTP clients, database pools, or Redis connections; set explicit connect and read timeouts on every client.

WHY:

Default timeouts in many frameworks (like Java's HttpURLConnection) are infinite. A slow dependency holding sockets open freezes application worker threads until the entire service crashes.

FIX:

Set connect timeout short (500ms - 1s), read timeout realistically tight (2s - 5s), and configure global request deadlines to prevent cascading thread starvation.

TIMEOUT BOUNDARY HIERARCHYSCRIBBLE BLUEPRINT
Layered timeout configurations preventing thread pool starvation across dependencies
#1

Upstream Caller

Global Deadline: 5.0s

#2

API Gateway

Request Timeout: 4.0s

#3

Your Microservice

App Processing: < 500ms

#4
Strictly Bounded

Downstream Call

Connect: 500ms | Read: 2.0s

#5

Slow Partner API

Hangs indefinitely (120s)

Execution Sequence & Transitions
Upstream CallerAPI Gateway[deadline=5s]
API GatewayYour Microservice[timeout=4s]
Your MicroserviceDownstream Call[enforce 2.5s total]
Downstream CallSlow Partner API[SocketTimeout trips at 2.0s!]
#Connect Timeout#Read Timeout#Cascading Failure#Thread Exhaustion#HTTP Client
Includes Code & Trade-off MatrixRead Article
Resilience & RetriesCRITICAL
8 min read
RULE:

Isolate malformed or crash-inducing messages ('poison pills') instantly so they cannot halt consumer loops or take down worker pods.

WHY:

A single corrupted message that causes an OutOfMemoryError, infinite loop, or uncaught deserialization exception will crash consumer pods in an endless restart cycle.

FIX:

Use error-handling deserializers (`ErrorHandlingDeserializer`), isolate workloads with Bulkhead patterns, and route poison pills directly to quarantine without retrying.

POISON PILL QUARANTINE SHIELDSCRIBBLE BLUEPRINT
Deserialization error-handling catching corrupted payloads before consumer pod crashes
#1
Poison Pill

Kafka Partition

Offset 402: Malformed JSON

#2

ErrorHandlingDeserializer

Catches parse exception

#3

Failure Classifier

Is transient or fatal?

#4
Zero Pod Crashes

Quarantine DLQ

orders.poison-pills + Alert

#5

Offset 403 (Healthy)

Processes successfully

Execution Sequence & Transitions
Kafka PartitionErrorHandlingDeserializer[poll message]
ErrorHandlingDeserializerFailure Classifier[deserialization error]
Failure ClassifierQuarantine DLQ[fatal: route immediately]
Failure ClassifierOffset 403 (Healthy)[advance partition offset]
#Poison Pill#Kafka#Deserialization Error#Bulkhead#DLQ
Includes Code & Trade-off MatrixRead Article
Data & State ConsistencyCRITICAL
8 min read
RULE:

Never run bare in-memory `@Scheduled` cron jobs across multi-instance deployments; always protect them with distributed locks and build shadow reconciliation loops.

WHY:

If your app runs 5 replicas, a bare cron job fires 5 times concurrently, sending duplicate billing invoices, charging customers 5x, or producing race conditions.

FIX:

Use ShedLock or Redis distributed locks with safety hold times, make jobs chunked and idempotent, and build daily reconciliation loops comparing system balances.

DISTRIBUTED SHEDLOCK & AUDITSCRIBBLE BLUEPRINT
ShedLock coordinating multi-replica cron execution + automated reconciliation audit
#1

App Replica 1

Cron triggers at 02:00

#2

App Replica 2

Cron triggers at 02:00

#3
Lock: 15m safety

Distributed Lock Table

shedlock table in DB / Redis

#4

Node 1 Wins Lock

Executes reconciliation chunk

#5

Node 2 Skips

Already locked -> Exit cleanly

#6

Reconciliation Ledger

Flag discrepancies > 0

Execution Sequence & Transitions
App Replica 1Distributed Lock Table[acquireLock('billing_recon')]
App Replica 2Distributed Lock Table[acquireLock('billing_recon')]
Distributed Lock TableNode 1 Wins Lock[GRANTED to Node 1]
Distributed Lock TableNode 2 Skips[DENIED to Node 2]
Node 1 Wins LockReconciliation Ledger[compare Internal DB vs Gateway]
#ShedLock#Distributed Cron#Reconciliation Loop#Idempotent Jobs#Ledger Audit
Includes Code & Trade-off MatrixRead Article
Ops, Config & SecurityCRITICAL
7 min read
RULE:

Never commit plaintext secrets or config files to Git; inject production configs dynamically through secure secret stores with strict role-based access.

WHY:

Exposed API keys, database credentials, and production toggle endpoints in source code lead to total infrastructure compromise and data breaches.

FIX:

Use HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets with envelope encryption. Gate production runtime config updates behind strict multi-factor auth and audit logs.

ZERO-TRUST SECRET INJECTIONSCRIBBLE BLUEPRINT
Secure runtime injection from KMS/Vault into ephemeral pod memory without Git exposure
#1
Gitleaks Scan

Git Repository

Code ONLY (Zero Secrets)

#2

CI/CD Pipeline

Builds immutable container image

#3
Strict RBAC

HashiCorp Vault / KMS

Encrypted secrets & dynamic creds

#4

Production Pod

Injects into in-memory ENV at boot

#5
Never Public

Actuator / Config Endpoints

Protected behind VPN / Internal Auth

Execution Sequence & Transitions
Git RepositoryCI/CD Pipeline[commit]
CI/CD PipelineProduction Pod[deploy container]
HashiCorp Vault / KMSProduction Pod[mTLS secret lease]
Production PodActuator / Config Endpoints[internal only]
#Vault#AWS Secrets Manager#Zero Trust#Dynamic Config#RBAC
Includes Code & Trade-off MatrixRead Article
Ops, Config & SecurityESSENTIAL
7 min read
RULE:

Build a balanced testing pyramid: fast unit tests for domain logic, containerized integration tests (Testcontainers) for data access, and selective E2E tests.

WHY:

Relying solely on unit tests with mocked databases creates a false sense of security; tests pass locally, but fail in production on real SQL dialect differences.

FIX:

Test business rules in pure unit tests without Spring context (sub-millisecond). Test real queries against real PostgreSQL/Redis instances using Testcontainers in CI.

THE PRAGMATIC TESTING PYRAMIDSCRIBBLE BLUEPRINT
Ratio of tests, execution speed, and fidelity across testing layers
#1
Top 5 User Flows

10% E2E Smoke Tests

Playwright / Cypress on staging (Slow)

#2
High Fidelity

20% Integration Tests

Testcontainers (Real Postgres & Kafka)

#3
Instant Feedback

70% Pure Unit Tests

Pure domain logic & mockito (Sub-second)

Execution Sequence & Transitions
70% Pure Unit Tests20% Integration Tests[verify contracts]
20% Integration Tests10% E2E Smoke Tests[verify end-to-end journey]
#Testcontainers#Integration Tests#Unit Tests#Mocking#E2E Testing
Includes Code & Trade-off MatrixRead Article
Resilience & RetriesCRITICAL
7 min read
RULE:

Do not let a slow non-critical dependency fail your Kubernetes liveness probe; degrade gracefully and shed non-essential load.

WHY:

If your `/health/liveness` check verifies database connectivity and the database encounters a 5-second lock, Kubernetes kills all application pods simultaneously, causing a full cascading outage.

FIX:

Liveness probe: verifies only if process thread is alive (internal deadlock check). Readiness probe: verifies ability to accept traffic. Degrade non-critical features (recommendations, analytics) to keep core flows working.

K8S PROBES & DEGRADATIONSCRIBBLE BLUEPRINT
Distinction between internal liveness, network readiness, and feature degradation
#1

Kubernetes Kubelet

Probes every 10s

#2
Never check DB here!

Liveness Probe

Checks process deadlock ONLY

#3
Removes from Traffic

Readiness Probe

Checks DB/Kafka connection

#4

App Service Pod

Serving live traffic

#5

Degradation Fallback

Serve cached or static default

Execution Sequence & Transitions
Kubernetes KubeletLiveness Probe[HTTP /health/live]
Kubernetes KubeletReadiness Probe[HTTP /health/ready]
Readiness ProbeApp Service Pod[pass: route ingress traffic]
App Service PodDegradation Fallback[recs fail -> return trending items]
#Liveness Probe#Readiness Probe#Load Shedding#Kubernetes#Degradation
Includes Code & Trade-off MatrixRead Article
Ops, Config & SecurityESSENTIAL
7 min read
RULE:

Decouple code deployment from feature release using dynamic feature flags with automated kill switches.

WHY:

Deploying new code straight to 100% of users without flags means any bug requires an emergency rollback, re-building images, and re-deploying.

FIX:

Use percentage-based canary rollouts (1% -> 5% -> 25% -> 100%), test in production with dark launches, and instrument emergency kill switches that update in < 5 seconds without redeploying.

CANARY FEATURE ROLLOUTSCRIBBLE BLUEPRINT
Percentage-based deterministic hashing evaluation with instant kill switch
#1

User Request

User ID: 94821

#2

Flag Evaluator

hash(userId + flagKey) % 100

#3
10% Rollout

New V2 Algorithm

Bucket < 10% (Canary)

#4

Legacy V1 Algorithm

Bucket >= 10% (Stable)

#5
Kill Switch

Emergency Kill Switch

Instant OFF in < 2s without deploy

Execution Sequence & Transitions
User RequestFlag Evaluator[evaluate flag]
Flag EvaluatorNew V2 Algorithm[user in canary bucket]
Flag EvaluatorLegacy V1 Algorithm[default user]
Emergency Kill SwitchLegacy V1 Algorithm[override on anomaly]
#Feature Flags#Canary Rollout#Kill Switch#Dark Launch#LaunchDarkly
Includes Code & Trade-off MatrixRead Article
Observability & LoggingCRITICAL
8 min read
RULE:

Structure service metrics using the RED method (Rate, Errors, Duration) and alert on symptoms that impact users (SLO burn rate), not raw CPU spikes.

WHY:

Alert fatigue causes on-call engineers to ignore alarms. Waking up an engineer because CPU hit 85% on a batch worker is noise; alerting because checkout 5xx rate crossed 1% saves revenue.

FIX:

Instrument the RED method (req/sec, 5xx error rate, p95/p99 latency) on every API endpoint. Route alerts to PagerDuty only when customer-facing SLOs are breached.

THE RED METHOD & ALERTINGSCRIBBLE BLUEPRINT
Telemetry flow: App Prometheus metrics -> Grafana dashboards -> Actionable SLO alerts
#1

Application Traffic

HTTP Endpoints & Workers

#2

RED Telemetry

Rate, 5xx Errors, p99 Latency

#3

Prometheus TSDB

Scrapes /metrics every 15s

#4
Multi-Window Burn

SLO Rule Evaluator

99.9% availability error budget

#5
Page ONLY on impact

PagerDuty (On-Call)

Actionable alert + Runbook link

Execution Sequence & Transitions
Application TrafficRED Telemetry[Micrometer metrics]
RED TelemetryPrometheus TSDB[pull scrape]
Prometheus TSDBSLO Rule Evaluator[evaluate burn rate]
SLO Rule EvaluatorPagerDuty (On-Call)[SLO budget exhausted!]
#Prometheus#Grafana#RED Method#SLO / SLI#PagerDuty Alert Fatigue
Includes Code & Trade-off MatrixRead Article
API Contracts & ErrorsESSENTIAL
7 min read
RULE:

Treat APIs as immutable public contracts; maintain authoritative OpenAPI specifications and enforce strict backward compatibility.

WHY:

Renaming a field or changing a data type in an undocumented API silently breaks mobile apps and partner integrations, requiring emergency rollbacks.

FIX:

Use OpenAPI 3.1 / Swagger as the single source of truth. Apply additive changes only (never remove or rename existing fields). Use URL path versioning (`/v1`, `/v2`) for breaking paradigm shifts.

CONTRACT-FIRST LIFECYCLESCRIBBLE BLUEPRINT
OpenAPI single source of truth generating server stubs, SDKs, and CI breaking checks
#1
Source of Truth

OpenAPI 3.1 Spec

openapi.yaml in git

#2
Block PR if breaking

CI Breaking Check

oasdiff detects breaking changes

#3

Server Stubs

Generated Java/TS interfaces

#4

Client SDKs

iOS, Android & Frontend clients

#5

Interactive Docs

Swagger UI / Redoc

Execution Sequence & Transitions
OpenAPI 3.1 SpecCI Breaking Check[git push]
OpenAPI 3.1 SpecServer Stubs[code-gen]
OpenAPI 3.1 SpecClient SDKs[code-gen]
OpenAPI 3.1 SpecInteractive Docs[render]
#OpenAPI 3.1#Swagger#Contract-First#Backward Compatibility#SemVer
Includes Code & Trade-off MatrixRead Article
Ops, Config & SecurityESSENTIAL
6 min read
RULE:

Maintain strict separation between code and configuration; compile identical immutable container images and configure them via environment variables.

WHY:

Building environment-specific binaries (`my-app-dev.jar` vs `my-app-prod.jar`) introduces untested configuration differences and violates modern container principles.

FIX:

Follow Factor III of the 12-Factor App: store config in the environment. The exact same Docker image must run in local, dev, staging, and production without rebuilding.

12-FACTOR CONFIG DISCIPLINESCRIBBLE BLUEPRINT
One immutable container artifact configured purely via environment injections
#1
Build ONCE

Immutable Image

beingsde/api:v2.4.0 (Identical everywhere)

#2

Dev Environment

ENV: DB_URL=dev.internal:5432

#3

Staging Environment

ENV: DB_URL=stg.internal:5432

#4
Zero Rebuild

Production Cluster

ENV: DB_URL=prod-cluster:5432

Execution Sequence & Transitions
Immutable ImageDev Environment[deploy]
Immutable ImageStaging Environment[deploy]
Immutable ImageProduction Cluster[deploy]
#12-Factor App#Environment Parity#Config vs Code#Docker#Cloud Native
Includes Code & Trade-off MatrixRead Article
API Contracts & ErrorsESSENTIAL
7 min read
RULE:

Return consistent, structured error envelopes (RFC 7807 Problem Details); never leak raw stack traces or SQL exceptions to API clients.

WHY:

Raw stack traces expose internal class names, database schemas, and library versions to attackers, while inconsistent error shapes break frontend error handling.

FIX:

Standardize on RFC 7807: include `type`, `title`, `status`, `detail`, `instance`, and an opaque `error_id` (trace ID) so users can report issues without seeing raw code internals.

RFC 7807 ERROR ENVELOPESCRIBBLE BLUEPRINT
Standardized error payload protecting internal stack traces while providing client clarity
#1
Security Leak!

❌ RAW EXCEPTION LEAK

HTTP 500: PSQLException: relation 'users' does not exist at org.postgresql.core...

#2
Clean & Actionable

✅ RFC 7807 PROBLEM DETAILS

{ status: 422, title: 'Unprocessable Entity', detail: 'Insufficient stock', error_id: 'err_8f9a' }

#3

Customer Support / Triage

Look up error_id in Loki/Elasticsearch

Execution Sequence & Transitions
❌ RAW EXCEPTION LEAK✅ RFC 7807 PROBLEM DETAILS[sanitize via @ControllerAdvice]
✅ RFC 7807 PROBLEM DETAILSCustomer Support / Triage[quote error_id]
#RFC 7807#Problem Details#Error Envelopes#Security Leaks#HTTP Status
Includes Code & Trade-off MatrixRead Article
Data & State ConsistencyCRITICAL
8 min read
RULE:

Wrap all asynchronous messages in standardized Event Envelopes with strict schema versioning and deterministic event IDs.

WHY:

Publishing unstructured raw JSON payloads across queues leads to breaking contract changes, inability to replay events, and un-debuggable consumer crashes.

FIX:

Standardize on an Event Envelope (e.g. CloudEvents standard: `id`, `source`, `type`, `time`, `data_version`, `payload`). Enforce schema evolution with Avro or JSON Schema Registry.

EVENT ENVELOPE TOPOLOGYSCRIBBLE BLUEPRINT
Standardized CloudEvents envelope wrapping domain payload with schema validation
#1

Domain Action

Order Placed

#2
CloudEvents Spec

Event Envelope

id, source, type, timestamp, traceId

#3

Schema Registry

Validates backward compatibility

#4

Kafka Event Stream

orders.events.v1

#5

Idempotent Consumer

Deduplicates via event_id

Execution Sequence & Transitions
Domain ActionEvent Envelope[wrap payload]
Event EnvelopeSchema Registry[verify schema]
Schema RegistryKafka Event Stream[publish]
Kafka Event StreamIdempotent Consumer[deduplicate & process]
#Event-Driven#CloudEvents#Avro / Protobuf#Schema Registry#Pub/Sub
Includes Code & Trade-off MatrixRead Article
Database & PerformanceCRITICAL
9 min read
RULE:

Never release a database query without running `EXPLAIN ANALYZE`; eliminate N+1 queries and replace offset pagination with keyset pagination.

WHY:

Unindexed full table scans lock CPU cores, while offset pagination (`OFFSET 1000000`) forces the database to read and discard a million rows on every click.

FIX:

Index columns in foreign keys and `WHERE`/`ORDER BY` clauses following the Leftmost Prefix Rule. Use `JOIN FETCH` to eliminate N+1 queries and keyset pagination (`WHERE id > :lastId`) for large datasets.

INDEX OPTIMIZATION ANATOMYSCRIBBLE BLUEPRINT
Transforming slow Sequential Scans into sub-millisecond B-Tree Index Scans
#1
Disk I/O Bottleneck

❌ Seq Scan (10M Rows)

Execution: 4,820ms | CPU: 100%

#2

B-Tree Composite Index

CREATE INDEX idx_user_created ON orders (user_id, created_at DESC)

#3
10,000x Faster

✅ Index Scan (0.4ms)

Execution: 0.4ms | Zero Seq Scan

#4

N+1 Query Killer

JOIN FETCH / BatchSize = 50

Execution Sequence & Transitions
❌ Seq Scan (10M Rows)B-Tree Composite Index[add index based on EXPLAIN]
B-Tree Composite Index✅ Index Scan (0.4ms)[fast B-Tree lookup]
✅ Index Scan (0.4ms)N+1 Query Killer[eager load associations]
#EXPLAIN ANALYZE#B-Tree Index#Composite Index#N+1 Problem#Keyset Pagination
Includes Code & Trade-off MatrixRead Article
API Contracts & ErrorsESSENTIAL
7 min read
RULE:

Never catch `Exception` and silently swallow it; translate technical exceptions into rich domain exceptions at integration boundaries.

WHY:

Swallowing exceptions (`catch (Exception e) {}`) hides corrupted state, makes bugs impossible to locate, and causes silent data loss.

FIX:

Catch specific exceptions, wrap in custom domain exceptions with context, preserve original cause (`throw new DomainEx('failed', cause)`), and handle globally via `@ControllerAdvice`.

EXCEPTION TRANSLATION LADDERSCRIBBLE BLUEPRINT
Layered exception handling translating low-level driver errors into clean domain exceptions
#1

Low-Level Driver

Throws PSQLException (Deadlock)

#2

Integration Adapter

Catches & translates with cause

#3
Domain Context

Domain Service

Throws LockAcquisitionException

#4
Clean Boundary

Global @ControllerAdvice

Logs traceId & returns HTTP 409

Execution Sequence & Transitions
Low-Level DriverIntegration Adapter[raw driver error]
Integration AdapterDomain Service[wrap with cause]
Domain ServiceGlobal @ControllerAdvice[bubble up]
Global @ControllerAdviceLow-Level Driver[log error with traceId]
#Exception Handling#Domain Exceptions#ControllerAdvice#Anti-Pattern#Clean Code
Includes Code & Trade-off MatrixRead Article
Database & PerformanceCRITICAL
8 min read
RULE:

Define automated data retention and archival policies from day one; tier old data from expensive NVMe tables into cold S3 storage.

WHY:

A 5-year-old table with 500 million records causes slow backups, expensive cloud disks, sluggish indexes, and potential GDPR compliance fines.

FIX:

Partition large tables by date range (monthly/yearly), detach old partitions instantly without table locking, and offload to compressed Parquet files on S3 Glacier.

STORAGE TIERING LIFECYCLESCRIBBLE BLUEPRINT
Tiering hot transactional data to warm S3 Parquet and cold Glacier archive
#1
Sub-ms Queries

HOT: PostgreSQL NVMe

0 - 90 Days (Active OLTP)

#2
Instant (Zero Lock)

Partition Rotation

ALTER TABLE DETACH PARTITION

#3

WARM: S3 Iceberg/Parquet

90 - 365 Days (Athena/Analytics)

#4
$0.00099 / GB

COLD: S3 Glacier

> 1 Year (Legal & Tax Compliance)

Execution Sequence & Transitions
HOT: PostgreSQL NVMePartition Rotation[monthly cron]
Partition RotationWARM: S3 Iceberg/Parquet[export to Parquet]
WARM: S3 Iceberg/ParquetCOLD: S3 Glacier[S3 Lifecycle rule after 365d]
#Data Archival#Table Partitioning#S3 Glacier#TTL#GDPR Compliance
Includes Code & Trade-off MatrixRead Article