01. Monitoring / Metrics System — High-Level Design¶

~18 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, then follow a sample from collection to queryable and a dashboard query back out, then watch what happens when a bad deploy triples the series count.

Architecture¶

  targets / agents                          engineers / dashboards / alertmgr
        │  push (remote-write)                        ▲
        │  or scraped (pull)                          │ query / alerts
        ▼                                             │
  ┌───────────────┐                          ┌────────────────┐
  │ Ingest gateway│  auth, per-tenant rate   │ Query frontend │  split, cache,
  │ / distributor │  + cardinality limits    │                │  dedupe
  └──────┬────────┘                          └───────┬────────┘
         │ hash(series) → shard, replicate ×3        │ fan-out by time range
   ┌─────┼─────────────┐                     ┌───────┼─────────────┐
   ▼     ▼             ▼                     ▼                     ▼
┌────────┐ ┌────────┐ ┌────────┐      ┌──────────────┐   ┌──────────────────┐
│Ingester│ │Ingester│ │Ingester│◀─────│  Querier     │──▶│ Store-gateway    │
│ head + │ │ head + │ │ head + │ recent│ (PromQL eval)│   │ (reads blocks    │
│  WAL   │ │  WAL   │ │  WAL   │ series└──────────────┘   │  from object svc)│
└───┬────┘ └───┬────┘ └───┬────┘                          └────────┬─────────┘
    │ flush 2h blocks     │                                        │
    └─────────┬───────────┘                                        │
              ▼                                                     ▼
      ┌──────────────┐   downsample/merge   ┌──────────────────────────────┐
      │  Compactor   │─────────────────────▶│  Object storage (S3/GCS)     │
      │ (rollups)    │                       │  immutable blocks: index +   │
      └──────────────┘                       │  chunks + meta, tiered TTLs  │
                                             └──────────────────────────────┘

  ┌──────────────┐  evaluates 50k rules / 15s, over recent series
  │  Ruler       │───────────────────────────────▶ Alertmanager (dedupe,
  │ (alert eval) │      (queries ingesters/queriers) silence, route, page)
  └──────────────┘

Read it top to bottom. Samples enter at the ingest gateway, which authenticates the source, enforces per-tenant rate and cardinality limits, and hashes each sample by its series identity to pick a shard. Each sample is written to three ingesters for durability. Ingesters hold recent data in an in-memory head plus a write-ahead log, and every couple of hours flush an immutable block to object storage. The compactor merges those blocks and produces the downsampled rollups. On the read side, the query frontend splits and caches queries and hands them to queriers, which gather recent series straight from ingesters and older series from store-gateways that read blocks out of object storage, then run the PromQL evaluation and merge. The ruler runs the same query engine on a 15-second clock to evaluate alert rules and ships firing alerts to Alertmanager, which deduplicates, silences, and routes them to humans. The write path and the read path share only the object store and the ingesters' recent data — they scale independently.

Components¶

Ingest gateway / distributor. The front door for both push and pull. It authenticates and identifies the tenant, applies rate and cardinality limits (the single most important control in the system), computes a stable hash of each series' label set, and routes the sample to the ingester shards that own that series. It replicates each write to a replication factor of three and considers the write successful on a quorum of two acks, so a single ingester being down never stalls ingestion.

Ingester. Where live data lives. Each ingester owns a shard of the series space and keeps their recent samples in an in-memory head block — one open, compressed chunk per series — backed by a write-ahead log on local SSD so a crash loses only the last few seconds. Every ~2 hours it cuts the head into an immutable block, writes it to object storage, and truncates the WAL. Ingesters also answer queries for recent data directly, since that data has not yet reached object storage.

Compactor. A background service that takes the raw 2-hour blocks and does two jobs: vertical compaction merges the replicas and overlapping blocks into one deduplicated block, and downsampling produces the 1-minute and 1-hour rollups by aggregating each window into a fixed set of values (sum, count, min, max, last). It also enforces retention by deleting blocks past their TTL. This is the component that turns 62 TB/year into ~1.3 TB.

Object storage. The durable, cheap, effectively infinite system of record for everything older than the head. Blocks are self-describing: each carries its own inverted index, its compressed chunks, and a metadata file naming its time range and resolution. Storing the index inside each block is what lets store-gateways serve queries without a central index server.

Store-gateway. The read-side counterpart to the ingester. It knows which blocks in object storage cover which time ranges, caches their indexes, and on a query fetches only the postings and chunks the query needs — a few range reads against S3 rather than a scan.

Querier. The PromQL engine. Given a query and time range, it asks the ingesters for recent series and the store-gateways for older ones, deduplicates the overlap, intersects postings lists to find matching series, streams their chunks, and runs the aggregation and functions. Queriers are stateless and scale with query load.

Query frontend. Sits in front of queriers to make dashboards cheap: it splits a long range query into per-day sub-queries (so they parallelize and cache cleanly), serves a results cache for time ranges that have already been computed and can no longer change, and rate-limits or queues expensive queries so one bad dashboard cannot starve the tier.

Ruler. Runs alert and recording rules on a fixed cadence using the same query path as a querier. For each rule it evaluates the expression, tracks how long the condition has held (the for duration), and emits an alert to Alertmanager when the condition has been true long enough. Rules are sharded across ruler replicas by rule group.

Alertmanager. Takes the raw firing alerts and turns them into the right number of notifications: it deduplicates identical alerts from replicated rulers, groups related alerts, applies silences and inhibition, and routes to the right destination (page, chat, email). It is the piece that stands between "1,000 hosts breached a threshold" and "one useful page."

Primary write path (a sample becomes queryable)¶

A target exposes /metrics (pull) or an agent batches samples and POSTs to /api/v1/write (push). Either way a batch of (labels, ts, value) arrives at the ingest gateway.
The gateway authenticates the tenant and checks limits. If the tenant is over its series cardinality budget or ingest rate, the offending samples are rejected with 429 — better to drop new series than to let the fleet's index blow up.
For each sample the gateway hashes the series' label set to a series ID and looks up the three ingesters that own that shard, sending the sample to all three.
Each ingester finds (or, for a new series, creates) the series in its head, appends the sample to that series' open chunk with delta-of-delta + XOR compression, and writes the sample to its WAL. Creating a new series also inserts postings into the head's inverted index — the expensive case.
The gateway returns success once two of the three ingesters ack. The sample is now queryable from the ingesters' head; it will not appear in object storage for up to two hours.
Every ~2 hours each ingester flushes its head to an immutable block in object storage and truncates the WAL. Later the compactor merges the three replicas' blocks into one and, as the block ages past 48 hours and 30 days, produces the 1-minute and 1-hour rollups.

Primary read path (a dashboard range query)¶

A dashboard issues GET /api/v1/query_range with a PromQL expression, a start/end, and a step. It hits the query frontend.
The frontend checks the results cache; any sub-range already computed and now immutable (older than the head) is served from cache. It splits the remaining range into per-day chunks and dispatches them to queriers.
A querier parses the expression and extracts its label matchers. For time within the last couple of hours it queries the ingesters owning the matched shards; for older time it asks the store-gateways, which locate the covering blocks in object storage.
On each source, the label matchers are turned into an intersection of postings lists to yield the matching series IDs, and only those series' chunks are read — contiguous, compressed reads, no scan.
The querier deduplicates series that appear in both ingesters (replica 1, 2, 3) and store-gateways, decompresses the chunks, and runs the aggregation and functions (rate, sum by, percentiles) over the requested step.
Results merge back at the frontend, which caches the immutable portions and returns the matrix. Because old ranges are cached and can never change, the same dashboard reloaded costs almost nothing beyond its most recent, still-changing window.

Storage choices¶

Head (recent, hot): in-memory chunks + WAL on local SSD. Chosen because ingestion is an append that must never block; RAM gives microsecond appends and the WAL gives crash durability without a synchronous disk write per sample. The head holds roughly the last 2 hours, which is also where alerts and live dashboards read, so keeping it in memory serves the latency-critical reads for free.
Recent blocks: local SSD, columnar chunk format. Immutable 2-hour blocks with per-block inverted indexes, sorted by series then time so a query is a sequential read. Gorilla-style compression (delta-of-delta timestamps, XOR floats) puts a 10-second-interval sample near ~1.3–2 bytes.
Long-term blocks: object storage (S3/GCS). Cheap, durable, infinite, and the natural home for immutable blocks. The tradeoff is higher per-request latency, which is exactly why full-resolution data stays local for 48 hours and only downsampled blocks live here long-term.
Inverted index: postings lists, one per label pair, stored inside each block. label=value → sorted list of series IDs. Query-time label matching is a merge-intersection of these sorted lists. Co-locating the index with its block avoids a central index service that would itself have to scale to 10M series.
Alert state (the for timers): small replicated KV. Tiny compared to metric data but must survive ruler restarts, so it lives in a consistent store, not in ingester memory.

Scaling¶

Write path. Ingest scales by adding ingesters and rehashing the series space across the larger ring. At 1M samples/s across 30 ingesters, each carries ~33k samples/s and ~333k active series — well within a single node. The real scaling axis is not sample rate but series count: doubling active series from 10M to 20M doubles head memory from ~30 GB to ~60 GB (×3 for replication) and forces more ingesters regardless of sample rate. This is why the gateway's cardinality limit is a scaling control, not just a safety valve.

Read path. Queriers and store-gateways are stateless and scale horizontally with query volume. The query frontend's split-and-cache is the multiplier: splitting a 30-day dashboard query into 30 daily sub-queries lets 29 of them come from cache after the first load, so a heavily-watched dashboard costs one day's compute per refresh instead of thirty. Store-gateway index caching keeps object-storage requests to the handful of blocks and postings a query actually needs.

Partitioning. Series are hashed to shards by their full label set, which spreads both writes and the query fan-out evenly — no single ingester becomes hot from key skew, because a busy service's many series land on different shards. Time is the natural partition for storage (2-hour blocks), which is what makes retention a matter of deleting old block files.

Backpressure. When ingesters approach memory limits they shed load upward: the gateway returns 429, and well-behaved push clients buffer and retry with backoff while pull is simply skipped until the next interval. Dropping the newest samples under pressure is the correct failure — a 10-second gap heals on the next scrape, whereas an OOM-killed ingester loses everything in its head.

Operational signals¶

The healthy signal is active series count, which should track the fleet size and change only when the fleet does; a metrics backend at steady state has a flat series count. The first metric to degrade under trouble is ingest append latency / WAL write latency climbing on the ingesters, or the gateway's 429 rate ticking up — the write path feeling pressure before anything else. The misleading metric is samples-per-second ingested: it can look perfectly healthy and flat while active series quietly climbs, because appending to existing series is cheap — an operator watching only sample rate will miss a cardinality explosion entirely until the ingesters OOM. The graph an experienced operator opens first during an incident is active series over time, broken down by metric name and tenant: a sudden vertical line there localizes a cardinality blowup to the exact metric and team that shipped it, which is almost always the root cause when a metrics backend is in trouble.

Failure modes and resilience¶

Cardinality explosion (the threaded failure). A bad deploy adds a high-cardinality label — say user_id or a raw request_id — to a hot metric. Active series jumps from 10M toward 100M in minutes. Head memory rockets from ~30 GB toward ~300 GB per replica, ingesters approach OOM, and the inverted index churns so hard that appends to healthy series slow down too. This is why the gateway enforces a per-tenant series limit that rejects new series past a budget (returning 429 for the new ones while existing series keep flowing), why active series by metric is the first graph to open, and why the fix is a kill switch that drops the offending label at ingest until the deploy is rolled back. The samples were never the problem; the series were.
Ingester crash. The head is in memory, but the WAL on local SSD replays on restart, so only the last few seconds are at risk — and with replication factor 3, the other two replicas covered those seconds anyway. The ring rebalances the dead ingester's shard onto its peers until it returns.
Object storage slow or unavailable. Recent queries (last 2 hours) are unaffected because they read the ingesters' head. Historical queries degrade or fail, and block flushes queue in the ingesters — which extends WAL length and head memory, so a long object-store outage eventually pressures the write path. Mitigation: alert on flush backlog and keep enough local headroom to ride out a typical outage.
Query overload. A pathologically broad query ({__name__=~".+"} over 30 days) can try to load millions of series. The query frontend caps series touched and time-range span per query and queues expensive ones, so one abusive dashboard cannot starve alert evaluation, which shares the read path.
Ruler / Alertmanager loss. A dead ruler shard stops evaluating its rules — a silent gap in coverage, worse than a loud one. Rulers are replicated and Alertmanager runs as a gossiping cluster that deduplicates alerts from redundant rulers, so an alert fires once even when two rulers evaluate the same rule, and coverage survives a single node loss.
Clock skew and late samples. A source with a skewed clock or a network hiccup delivers samples out of order or minutes late. The ingester accepts within a bounded out-of-order window and rejects samples older than the head's oldest open chunk, so a badly-skewed source cannot corrupt already-flushed blocks.

Where this shows up in production¶

Facebook Gorilla / Beringei — pioneered the delta-of-delta + XOR float compression that gets 10-second samples down to ~1.37 bytes, the reason full-resolution recent data fits in memory at all.
Prometheus — the reference pull model with an in-memory head, a WAL, 2-hour immutable blocks, and per-block inverted indexes; the local single-node shape this study scales out.
Grafana Mimir / Cortex — the horizontally-scaled version: distributor, replicated ingesters, compactor, store-gateway, query frontend against object storage — the box diagram above is essentially theirs.
Datadog — the push model at scale: agents ship to an intake that enforces per-account cardinality billing, the commercial embodiment of "cardinality is the cost."
Uber M3 / M3DB — a purpose-built distributed TSDB with aggregation tiers, showing how downsampling is run as a first-class ingestion-time rollup, not only a background compaction.
Google Monarch — an in-memory, regional, push-based system prioritizing ingest availability over strong consistency, the extreme statement of "protect the write path."
Thanos — bolts long-term object-storage retention and a global query view onto vanilla Prometheus, the canonical "store-gateway reads blocks from S3" pattern.
VictoriaMetrics — an alternative storage engine tuned for very high cardinality, illustrating that the whole competitive frontier of this product category is who handles new-series churn most cheaply.
Alertmanager (Prometheus) — the dedup/group/silence/route layer that turns thousands of raw firing alerts into a sane number of pages, the reason a fleet-wide breach is one incident, not ten thousand.