Design a Metrics Monitoring & Alerting System

Observability platforms ingest millions of metric data points per second from microservices, store them in time-series databases, evaluate real-time alert rules, and render low-latency query dashboards.

1. System Requirements

Functional Requirements

1. Metrics Ingestion: Collect counters, gauges, and histograms from distributed hosts and microservices (e.g. cpu_usage{host="srv-1"} 78.4 1672531199).
2. Time-Series Querying: Support fast range queries and aggregations across custom tag dimensions.
3. Alerting Rules: Continuously evaluate alerting thresholds (e.g. p99 latency > 500ms for 5m) and dispatch PagerDuty/Slack webhooks.
4. Dashboards: Visualize real-time graphs with sub-second page loads.

Non-Functional Requirements

1. Massive Write Throughput: Ingest 10 Million metric samples per second.
2. Storage Efficiency: Compress time-series data to $< 1.5\text{ bytes per data point}$ using delta-of-delta and Gorilla compression.
3. Query Latency: Dashboard queries over past 1 hour return in $< 200\text{ms}$.

2. Ingestion Model: Pull vs. Push

Pull Model (Prometheus Style)

• The central metrics engine periodically scrapes an /actuator/prometheus HTTP endpoint exposed by each microservice.
• Pros: Simple client implementation; easy detection of down/unreachable nodes.
• Cons: Requires service discovery mechanism (Consul/Kubernetes DNS); struggles with ephemeral serverless functions (AWS Lambda).

Push Model (Datadog Style)

• Lightweight agents running on every server stream UDP/gRPC packets to a local forwarder or central collector.
• Pros: Natural fit for dynamic/ephemeral workers; supports high-frequency telemetry.
• Cons: Influx spikes can overwhelm ingestion servers unless cushioned by an upstream Kafka buffer.

3. High-Level Architecture Blueprint

                          +-----------------------+
                          |   Monitored Services  |
                          +-----------+-----------+
                                      |
                                      | gRPC / OTLP Metrics Stream
                                      v
                          +-----------------------+
                          |   Ingestion Gateway   |
                          +-----------+-----------+
                                      |
                                      v
                          +-----------------------+
                          | Kafka Metrics Buffer  |
                          +-----+-----------+-----+
                                |           |
       +------------------------+           +------------------------+
       | Raw Metrics Stream                 | Alert Evaluation Stream
       v                                    v
+------------------------+           +------------------------+
| Time-Series DB (TSDB)  |           | Alerting Rule Engine   |
| (Gorilla Compression / |           | (Sliding Window Engine)|
| ClickHouse / InfluxDB) |           +-----------+------------+
+------------------------+                       |
                                                 v
                                     +------------------------+
                                     | PagerDuty / Slack Alert|
                                     +------------------------+

Key Architecture Trade-Offs & Decision Matrix