02. Monitoring / Metrics System — Low-Level Design¶

~22 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)

The HLD named the boxes. This file opens the three that carry the design's weight — the ingester head with its chunk encoder, the query engine's postings intersection, and the ruler's alert evaluation — and pins down the data formats, the compression and downsampling math, and the concurrency corners where a metrics backend actually breaks.

Data models¶

Every measurement is a sample belonging to a series, and a series is fully identified by its sorted set of labels. The metric name is itself a label, __name__.

series identity  =  sorted labels  →  hashed to a stable 64-bit series ID
  {__name__="http_requests_total", service="checkout", region="us-east", status="500"}
    → series_id = 0x9f3a...   (fnv/xxhash over the canonical label string)

sample           =  (series_id, timestamp_ms:int64, value:float64)

The series ID is derived from the labels, not assigned by a counter, so any ingester or querier computes the same ID for the same series without coordination — this is what lets the gateway route by hashing and lets replicas agree on identity. Two non-obvious consequences: labels must be sorted canonically before hashing (otherwise {a,b} and {b,a} become different series), and label value is part of identity, so status="500" and status="503" are distinct series — which is exactly the mechanism a cardinality explosion abuses.

The inverted index maps each label pair to the sorted series IDs carrying it:

postings:   "service=checkout"  → [ 0x0012, 0x0033, 0x9f3a, ... ]   (sorted asc)
            "status=500"        → [ 0x0007, 0x9f3a, 0xA1B2, ... ]
            "__name__=http_requests_total" → [ ... ]

Sorted lists are the whole trick: a query's label matchers become an intersection of sorted lists, computable in a linear merge. There is one postings list per distinct label pair, and the count of series (not samples) is what sizes this index — 10M series is the number that matters.

A chunk holds one series' samples for a bounded window, compressed:

chunk = {
  series_id,
  min_ts, max_ts,           # time bounds, for query pruning
  encoding = XOR,           # Gorilla: dod timestamps + XOR floats
  num_samples,
  bytes[]                   # the compressed stream (see algorithm below)
}

An immutable block on disk or in object storage bundles chunks, index, and metadata so it is self-contained:

block/
  meta.json     # ULID, min/max time, resolution (raw|1m|1h), source, stats
  index         # the inverted index (postings) + series→chunk offsets, for THIS block
  chunks/       # concatenated compressed chunks, sorted by (series_id, min_ts)
  tombstones    # deleted-series markers (deletes are rare, done at compaction)

Downsampled blocks carry the same shape but each "sample" is an aggregate tuple, so range/rate functions can be answered without the raw points:

rollup sample @ 1m/1h resolution:
  (bucket_ts, count, sum, min, max, last)
  # avg = sum/count; rate/increase use first & last of counter series

Component internals¶

Component 1 — Ingester head + chunk encoder¶

Responsibility: accept a million appends a second, keep recent data queryable in memory, survive a crash, and flush immutable blocks.

class Head:
    def append(series_id, ts, value) -> AppendResult   # hot path, per sample
    def get_or_create(labels) -> series_id             # index mutation on new series
    def query(matchers, min_ts, max_ts) -> Iterator[series]
    def flush() -> Block                                # every ~2h, cuts immutable block
    def replay_wal() -> None                            # on restart

class ChunkEncoder:                                     # one open chunk per series
    def append(ts, value) -> None                       # delta-of-delta + XOR
    def bytes() -> bytes

The head keeps a map from series_id to its open ChunkEncoder plus the in-memory postings index. append is the path that must stay cheap: look up the encoder (a hash-map hit for an existing series), append to its compressed stream, and append the raw sample to the WAL buffer. Only get_or_create for a new series is expensive — it allocates a series, inserts it into every relevant postings list (an index mutation under a lock), and is the operation the cardinality limiter guards. When an encoder's chunk reaches ~120 samples or the head ages past 2 hours, the chunk is closed and a new one opened; at the 2-hour mark the whole head is flushed to a block and the WAL truncated.

Component 2 — Query engine (postings intersection + chunk merge)¶

Responsibility: turn a PromQL expression into the minimal set of series reads, across ingesters and store-gateways, and evaluate the functions.

class Querier:
    def select(matchers, min_ts, max_ts) -> SeriesSet   # postings intersection
    def eval(expr, start, end, step) -> Matrix

def intersect(postings_lists) -> sorted series_ids      # linear k-way merge
def dedupe(replica_series) -> series                    # RF=3 overlap + store overlap

select is where flexibility is bought back: each matcher (service="checkout", status=~"5..") resolves to one or more postings lists, and the engine intersects them. Because the lists are sorted, intersection is a linear walk with no random access — the reason the index scales. Regex matchers (status=~"5..") expand to the union of matching label values' lists first, then intersect. Only after the matching series IDs are known does the engine read chunks, and only for those series, pruning by each chunk's min_ts/max_ts against the query window. Because the same series can arrive from three ingester replicas and from store-gateway blocks, dedupe merges them, preferring the source with the most complete, freshest samples.

Component 3 — Ruler (alert evaluation with `for` state)¶

Responsibility: evaluate 50,000 rules every 15 seconds and fire an alert only after its condition has held for the rule's for duration.

class Ruler:
    def evaluate(rule, now) -> AlertState                # runs rule.expr as a query
    # per (rule, output series) it tracks: {state: inactive|pending|firing, active_since}

# each tick, for each rule:
#   result = query(rule.expr, at=now)
#   for each series in result crossing the threshold:
#       if state == inactive:  state=pending; active_since=now
#       if now - active_since >= rule.for:  state=firing → emit to Alertmanager
#   for series no longer crossing:  state=inactive (resolve)

The for duration is what suppresses flapping: a rule like rate(errors[5m]) > 100 for: 2m only pages when the condition has been continuously true for two minutes, so a single scrape spike does not wake anyone. The pending/firing state per output series must survive a ruler restart (see edge cases), so active_since is checkpointed to the small consistent store from the HLD rather than held only in memory.

Core algorithm — Gorilla compression, on the threaded numbers¶

The reason 10M series at 10-second cadence fits in a manageable memory and byte budget is the delta-of-delta + XOR encoding. Walk it on our scenario, where a series emits a value every 10,000 ms.

Timestamps — delta-of-delta. Regular 10-second samples produce a near-constant delta, so encode the change in the delta:

Store the first timestamp fully (e.g. 1720008000000), and the first delta as-is (10000 ms).
For each later sample, compute dod = (t_n − t_{n−1}) − (t_{n−1} − t_{n−2}). For a perfect 10s cadence dod = 0.
Encode dod = 0 in a single bit (0). Small jitter (a scrape landing at 10,003 ms) costs a handful of bits via a variable-length bucket; only a large gap costs a full word.

Because our samples arrive every 10 seconds like clockwork, the overwhelming majority of timestamps cost 1 bit instead of 64 — a 64× reduction on the timestamp half before we touch the values.

Values — XOR. Most metrics change slowly or not at all between adjacent samples (a gauge holding steady, a counter ticking up by small amounts):

Store the first value as a full 64-bit float.
For each later value, xor = value_n XOR value_{n−1}. If the value is unchanged, xor = 0 → store a single bit (0).
If it changed, store a control bit plus the meaningful (non-zero) bits of the XOR, reusing the leading/trailing-zero window of the previous XOR when it fits — so a small numeric change costs a dozen-odd bits, not 64.

Put the scenario through it: 86.4 billion samples/day. Naive at 16 bytes each is 1.38 TB/day. With most timestamps at 1 bit and most values at 1–2 bytes of XOR payload, the average lands near 1.3–2 bytes/sample, giving ~170 GB/day — the number the storage budget in the overview is built on. The compression is not incidental; it is what makes keeping recent data in RAM, and thus serving alerts and live dashboards from memory, financially and physically possible.

Downsampling — the second-order saving. After 48 hours the compactor rolls each series' raw chunks into 1-minute buckets. For each 60-second window it reads the ~6 raw samples and writes one rollup tuple (bucket_ts, count, sum, min, max, last). Storing one aggregate tuple per minute in place of six raw samples cuts the point rate ~6×, from 170 GB/day to ~28 GB/day. After 30 days a second pass rolls 1-minute buckets into 1-hour buckets (another ~60×, to ~0.5 GB/day). A query over last quarter automatically reads the 1-hour blocks — sum, rate, and avg are all reconstructable from (count, sum, min, max, last) — so the dashboard is fast and cheap without the querier or the user knowing which resolution answered it. Thirteen months lands at ~1.3 TB versus ~62 TB, the 48× the overview promised.

Sequence diagram — a sample's write path with replication¶

Agent      Gateway         Ingester-1     Ingester-2     Ingester-3      WAL(disk)
  │  write batch  │             │              │              │             │
  ├──────────────▶│ auth +      │              │              │             │
  │               │ cardinality │              │              │             │
  │               │ check (ok)  │              │              │             │
  │               ├─ hash(series)→ shard owners = {1,2,3}      │             │
  │               ├────────────▶│ append+WAL   │              │             │
  │               ├─────────────┼─────────────▶│ append+WAL   │             │
  │               ├─────────────┼──────────────┼─────────────▶│ append+WAL  │
  │               │             ├──────────────┼──────────────┼────────────▶│ fsync buf
  │               │◀── ack ──────┤              │              │             │
  │               │◀── ack ──────┼──────────────┤              │             │
  │               │  (2 of 3 → quorum reached)  │              │             │
  │◀── 200 ───────┤             │   (3rd ack arrives later, fine)            │
  │               │             │              │              │             │
  │        ...every ~2h...      ├─ flush head → immutable block → object store
  │               │             ├─ truncate WAL                              │

Quorum on two of three acks means one slow or dead ingester never stalls the write; the third catches up asynchronously, and the compactor later dedupes the three replicas' blocks into one.

Concurrency and edge cases¶

Duplicate samples (idempotency). A push client retrying after a timeout resends a batch. Identity is (series_id, timestamp), so an append whose (series_id, ts) already exists in the open chunk is a no-op if the value matches and an error if it differs — a monitoring backend treats a resent sample as idempotent, not as a second data point, so retries never double-count.
Out-of-order and late samples. Samples must generally arrive in timestamp order per series because the chunk encoder appends forward. A bounded out-of-order window (say a few minutes) is accepted by buffering; anything older than the head's oldest open chunk is rejected, because it would require rewriting an already-compressed or already-flushed chunk. A badly clock-skewed source thus loses its late samples rather than corrupting a sealed block.
New-series race under the cardinality limiter. Two appends for the same brand-new series can arrive at one ingester concurrently. get_or_create takes a per-shard lock (or CAS on the series map) so exactly one series and one set of postings entries is created; the limiter's counter is incremented in the same critical section, so two racing new series cannot both slip past a budget of one.
Replica divergence. Ingester 3 was briefly down and missed some samples that 1 and 2 have. Both versions flush blocks; the compactor's vertical merge unions the samples by (series_id, ts), so the merged block is the superset and the gap self-heals. Queries before compaction dedupe across replicas and prefer the most complete stream, so the gap is invisible to readers too.
Alert state across ruler restart. If a ruler holding a rule in pending (condition true for 90s of a required 120s) restarts, in-memory active_since would reset and the alert would never mature. Checkpointing active_since to the consistent store, and reloading it on startup, keeps the for timer honest across restarts and across a rule moving to a different ruler replica.
Duplicate alerts from replicated rulers. Two ruler replicas evaluate the same rule for high availability, so the same alert is emitted twice. Alertmanager deduplicates by the alert's label fingerprint in its gossiping cluster, so a page fires once. The tradeoff is deliberate: run rulers redundantly (risking duplicate emits, which Alertmanager collapses) rather than singly (risking a silent gap in alert coverage) — a missed alert is far worse than a deduped one.
Staleness markers. When a target disappears (scrape fails, or a pushed series stops), the ingester writes an explicit stale marker into the series so a query does not carry the last value forward forever. Without it, a dashboard would show a dead host's CPU frozen at its final reading, and an alert on that value would never clear — the stale marker makes "no data" distinguishable from "unchanged data."