Skip to content

00. Design a Monitoring / Metrics System

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A metrics system collects numeric measurements from every service, host, and device in a fleet, stores them as time series, and lets engineers query and alert on them. This is the product behind Datadog, Prometheus, and the internal systems at every large operator — Facebook's Gorilla, Uber's M3, Google's Monarch. A time series is a named stream of (timestamp, value) pairs tagged with labels: http_requests_total{service="checkout", region="us-east", status="500"} is one series, and changing any label value makes a different series. The job is to accept a firehose of these samples, keep them cheaply for months, answer dashboard queries over them in under a second, and fire alerts when a series crosses a threshold — all while the fleet it is watching grows and occasionally misbehaves.

What makes this a real system-design problem is the collision of three workloads that pull in opposite directions. Ingestion is a relentless append at enormous volume. Storage has to be cheap enough to keep a year of history without bankrupting the company. Queries need to slice arbitrary label combinations and aggregate across thousands of series fast enough to redraw a dashboard. A storage layout tuned for one of these is hostile to the others, and the design is mostly about refusing to let any one of them win outright.

Thread one scenario through the whole study: a fleet emitting 1,000,000 data points per second across 10,000,000 active time series, feeding dashboards that must stay responsive and alert rules that must fire within seconds of a breach. Those two numbers are not independent — ten million series receiving a million samples a second means each series reports once every ten seconds — and that ten-second cadence, multiplied out, is what forces every storage and indexing decision below.

Functional requirements

  • Ingest samples for millions of series: (series_labels, timestamp, value), arriving continuously from thousands of sources.
  • Store with configurable retention, at full resolution recently and downsampled (rolled-up) for the long tail, so a year of history stays affordable.
  • Query with a label-matching, aggregating query language (PromQL-style): filter by labels, aggregate across series, apply rate/percentile functions, over an arbitrary time range and step.
  • Alert: evaluate rules (each a query plus a threshold and a duration) on a fixed cadence and route firing alerts to humans with deduplication and silencing.
  • Dashboards: serve many concurrent range queries fast enough to feel live.

De-scoped for this round, and worth naming so the interviewer hears a choice rather than a gap: distributed tracing and log storage (different data shapes, different systems), the full alert-routing and on-call escalation product (paging, schedules), synthetic monitoring and RUM, and anomaly-detection ML. Each is a real Datadog product line; none changes the core metrics pipeline, which is what this study is about.

Non-functional requirements

The single dominant constraint is sustained write throughput without letting index cardinality or query load stall ingestion. A metrics backend that drops samples is blind exactly when an incident is unfolding, so the write path must never block on the read path or on a slow storage tier.

  • Ingest availability: samples must be accepted continuously; a brief query outage is survivable, a brief ingest outage means permanent holes in the record. Protect the write path first.
  • Query latency: dashboard range queries over recent data should return in well under a second (p99 a few hundred ms); heavy historical queries over months can take seconds, and that is acceptable.
  • Alert freshness: rules evaluate on a 15-second cadence, so a breach is detected within roughly one evaluation interval — call it under 30 seconds end to end including notification.
  • Durability with tolerance for tiny loss: losing a few seconds of samples on a crash is acceptable (the next scrape refills the picture); losing hours is not. This tolerance is what lets the write path stay fast.
  • Cost: retention is a line item. Full-resolution storage of everything for a year is financially out of the question, which is why downsampling is a requirement, not an optimization.

Scale estimation

Start from the two threaded numbers and derive the rest.

Ingest rate. 1,000,000 data points/second is the design input. Across 10,000,000 active series that is one sample per series every 10M / 1M = 10 seconds — a standard scrape interval. Per day that is 1,000,000 × 86,400 = 86.4 billion samples/day, and roughly 31.5 trillion samples/year. This is the number the whole storage design answers to.

Storage. A raw sample is a 16-byte pair (an 8-byte timestamp and an 8-byte float). Uncompressed, a day is 86.4B × 16 B ≈ 1.38 TB/day — untenable to keep. Time-series compression saves us: delta-of-delta encoding on timestamps that arrive every 10 seconds collapses each timestamp to about a bit, and XOR encoding on slowly-changing float values (the Gorilla scheme) brings the average sample to roughly 1.3–2 bytes. Take 2 bytes with index overhead: 86.4B × 2 B ≈ 170 GB/day at full resolution. Kept naively for 13 months that is 170 GB × 395 ≈ 62 TB — still enough money to make downsampling mandatory.

Downsampling reconciles it. Keep full 10-second resolution for 48 hours (~340 GB). Roll up to 1-minute resolution for the next ~28 days, cutting the point rate ~6× to ~28 GB/day (~800 GB for the month). Roll up again to 1-hour resolution for the following year, another ~60× to ~0.5 GB/day (~170 GB for the year). The 13-month total lands near 1.3 TB instead of 62 TB — roughly a 48× saving — while recent data, which is what dashboards and alerts actually read, stays at full fidelity.

Index memory. Ten million active series is the cost that bites, not the byte count. Each active series needs its labels and an open chunk resident to accept writes and be found by queries — call it ~3 KB per series including its inverted-index postings, so 10M × 3 KB ≈ 30 GB of RAM for the write-side index. Replicated 3× for durability, that is ~90 GB spread across the ingest tier. This is why the defining risk here is cardinality: the samples are cheap, but every new series is a permanent tax on memory.

Alerting. Say 50,000 alert rules, each a query over the last few minutes of matched series, evaluated every 15 seconds — about 50,000 / 15 ≈ 3,300 rule evaluations/second, each touching recent, in-memory data. Sharded across a ruler tier, this is comfortably parallel.

Bandwidth. Batched, compressed remote-write puts the steady sample payload in the low tens of MB/s — never the bottleneck. The bottleneck is always index churn and query fan-out, not the wire.

API sketch

# Ingest — push model (remote write): batched, compressed samples
POST /api/v1/write
  body: snappy-compressed protobuf of [ {labels, samples:[(ts,val)...]} ... ]
  200:  accepted   (429 on backpressure — client retries with backoff)

# Ingest — pull model: the backend scrapes this exposition endpoint
GET  /metrics
  200:  http_requests_total{service="checkout",status="500"} 42 1720008000

# Query — instant and range (range powers dashboards)
GET  /api/v1/query?query=<promql>&time=<ts>
GET  /api/v1/query_range?query=<promql>&start=<ts>&end=<ts>&step=<dur>
  200:  { "resultType":"matrix", "result":[ {metric:{...}, values:[[ts,val]...]} ] }

# Alerting
POST /api/v1/rules        { "expr":"rate(errors[5m]) > 100", "for":"2m", ... }
GET  /api/v1/alerts       # currently firing/pending alerts

Solutioning

The traffic shape dictates the storage engine. An append at a million samples a second, read back as contiguous time ranges per series, is exactly the workload a log-structured merge tree serves well and a B-tree serves badly. Samples for a series accumulate in an in-memory head chunk, get compressed with delta-of-delta timestamps and XOR'd floats, and are periodically flushed as immutable, time-bounded blocks — recent blocks on local SSD, old blocks on object storage. The reframing that makes the whole design click: ingesting a million points a second is not a write-throughput problem; it is an index-churn problem. Appending a value to a series that already exists is nearly free — you extend a chunk in memory. The expensive event is the first sample of a new series, because that mutates the inverted index that maps labels to series. Keep the series set stable and this system hums at a million writes a second on modest hardware; let series churn and it falls over at a fraction of that.

The first defining tradeoff is write throughput versus query flexibility. Flexible queries ("all status=500 series in region=us-east, rate over 5 minutes, summed by service") demand a rich inverted index over every label, which is precisely the structure that makes writes expensive. The resolution is to split the problem: store the samples in a dumb, append-optimized columnar chunk format that knows nothing about labels, and store the label-to-series mapping in a separate inverted index of sorted postings lists. A query first intersects postings lists to find the matching series IDs, then does contiguous chunk reads for just those series. Writes touch the index only on new series; reads touch it on every query. Two structures, each optimal for its side, joined by a series ID.

The second tradeoff is retention versus cost, resolved by downsampling. Nobody queries last March at 10-second resolution; they query it at hourly granularity to see a trend. So the system keeps full resolution only as long as anyone looks at it that closely — 48 hours here — then a background compactor rolls windows up into 1-minute and then 1-hour aggregates, storing a small fixed set of rollups (typically sum, count, min, max, and last) per bucket so most query functions can be answered from the rollup without the raw points. The memory hook: old data is not deleted, it is blurred — you trade resolution you no longer need for a ~48× cut in stored bytes, and queries transparently pick the coarsest resolution that still answers the question.

The third tradeoff is push versus pull collection, and it is a genuine fork with no universal winner. Pull (Prometheus: the backend scrapes each target's /metrics every 10 seconds) gives the backend control over rate and cardinality, makes a target's up/down state a free byproduct of the scrape, and needs no credentials flowing inbound — but it struggles with short-lived jobs and networks where the backend cannot reach targets. Push (Datadog: an agent ships samples to an intake endpoint) handles ephemeral workloads, serverless, and locked-down networks naturally, but hands cardinality control to the client, so a misconfigured emitter can flood the intake before the backend can say no. The pragmatic answer most large systems land on is pull for infrastructure you control, push for everything at the edge, and enforce per-tenant cardinality and rate limits at the ingest gateway regardless — because whichever way samples arrive, the thing that kills you is the same: uncontrolled new series.

The result is a system whose write path is an in-memory append plus a rare index update, whose storage tiers from SSD to object store and blurs with age, whose read path intersects postings then streams chunks, and whose alerting rides the same query engine on a fixed clock. The next file turns these into components; the one after that into chunk formats, posting-list intersection, and the concurrency corners where a metrics backend actually breaks.