03. Monitoring / Metrics System — Interview Q&A

~17 min read · Part 4 of 4 (Overview → HLD → LLD → Q&A)

These are the questions an interviewer asks once the pipeline is on the board. Each answer is the strong version, followed by the wrong answer that quietly sinks candidates.

Q1. Push or pull for collection — which and why? Neither is universally right; name the tradeoff and pick per source. Pull (scrape /metrics every 10s) gives the backend control over rate and cardinality, makes a target's up/down state a free byproduct of the scrape, and needs no inbound credentials — but it struggles with short-lived jobs and unreachable networks. Push (an agent ships to an intake) handles ephemeral and serverless workloads and locked-down networks, but hands cardinality control to the client, so a bad emitter can flood you. The senior answer is pull for infrastructure you control, push for the edge, and — this is the point — enforce per-tenant cardinality and rate limits at the ingest gateway either way, because the thing that kills you is the same regardless of direction: uncontrolled new series. Common wrong answer to avoid: "Push is modern, pull is legacy" (or the reverse). Treating it as fashion instead of a tradeoff over ephemerality and cardinality control misses the actual engineering question.

Q2. At 1,000,000 samples/second, how do you avoid drowning in storage? Two mechanisms, stacked. First, time-series compression: with samples arriving every 10 seconds, delta-of-delta timestamps collapse to about a bit and XOR'd float values to a byte or two, taking a naive 16-byte sample down to ~1.3–2 bytes — 86.4B samples/day becomes ~170 GB/day instead of 1.38 TB/day. Second, downsampling: keep full 10-second resolution only 48 hours, roll up to 1-minute for a month (~6× fewer points) and 1-hour for a year (~60× more), so 13 months lands near 1.3 TB instead of ~62 TB, a ~48× saving. Recent data — what alerts and live dashboards read — stays full-fidelity; only the old, coarsely-queried tail is blurred. Common wrong answer to avoid: "Store everything at full resolution in a big columnar warehouse and query it directly." The bytes bankrupt you and the query latency over a year of 10-second points is nowhere near dashboard-fast.

Q3. A bad deploy adds a user_id label to a hot metric and active series jumps from 10M toward 100M. What happens and what do you do? Recognize the category first: this is not a storage problem, it is an index-cardinality problem. The samples-per-second barely move — appending to series is cheap — but each new series is a permanent tax on the inverted index and head memory, which rockets from ~30 GB toward ~300 GB per replica, and the index churn slows appends to healthy series too, so the ingesters approach OOM. The immediate lever is the gateway's per-tenant series budget, which rejects the new series with 429 while existing series keep flowing, plus a kill switch that drops the offending label at ingest until the deploy is rolled back. The graph that localizes it in seconds is active series broken down by metric name and tenant — the offending metric is a vertical line. Common wrong answer to avoid: "Add more storage / scale up the database." Storage was never the bottleneck; you would burn money and still OOM the ingesters, because the cost is series count, not bytes.

Q4. How do you evaluate 50,000 alert rules in near real time without missing a breach? Run a sharded ruler tier that evaluates rules on a fixed 15-second clock — 50,000 / 15 ≈ 3,300 evaluations/second — each rule being a query over the last few minutes of matched series, which is resident in the ingesters' in-memory head, so evaluation is a fast in-RAM read, not a cold storage scan. Suppress flapping with each rule's for duration (fire only after the condition holds continuously for, say, 2 minutes), and checkpoint the for timer so a ruler restart does not reset a pending alert. Run rulers redundantly and let Alertmanager deduplicate, so a node loss never creates a silent gap in coverage. End to end a breach pages in under ~30 seconds. Common wrong answer to avoid: "Run alert queries against the long-term store / object storage." That adds seconds of latency per rule and couples alerting to the slowest tier; alerts must read the hot, in-memory recent data.

Q5. Why not just use PostgreSQL (or Cassandra) for the time series? Because the access pattern fights a general-purpose engine. A B-tree row store pays an index write per sample and stores 16+ bytes per point with no time-series compression, so a million inserts a second and a 48× storage blow-up sink it. Cassandra handles the write rate but its wide-partition model still lacks delta-of-delta/XOR compression and the label-postings inverted index that makes arbitrary label queries fast. A purpose-built TSDB splits samples (append-optimized, compressed columnar chunks) from the label index (sorted postings lists), which is exactly the structure a relational engine does not give you for free. Common wrong answer to avoid: "A database is a database, Postgres scales fine with partitioning." Partitioning helps time range pruning but does nothing for per-sample compression or label-set indexing, and the insert rate alone breaks it.

Q6. Talk me through the write-throughput-versus-query-flexibility tradeoff. Flexible queries — "all status=500 in region=us-east, rate over 5m, summed by service" — need a rich inverted index over every label, and maintaining that index is precisely what makes writes expensive. If you index everything eagerly on the write path, ingestion slows; if you index nothing, queries must scan. The resolution is to separate the two structures: samples go into a dumb append-only chunk format that knows nothing about labels, and a separate inverted index maps label pairs to sorted series IDs. Writes touch the index only when a new series appears (cheap at steady state); reads intersect sorted postings lists then do contiguous chunk reads for just the matching series. Each structure is optimal for its own side, joined by the series ID. Common wrong answer to avoid: "Index every label on write so queries are always fast." That collapses ingest throughput and turns every new-series event into a heavy write — the exact failure mode of a cardinality spike, made permanent.

Q7. How does downsampling work, and what do you lose? The compactor reads a window of raw samples and writes a fixed aggregate tuple per bucket — (bucket_ts, count, sum, min, max, last) — at 1-minute then 1-hour resolution. You keep the ability to compute avg (sum/count), min, max, and counter rate/increase (from last across buckets), so most dashboard functions are unaffected. What you lose is within-bucket detail: a 3-second latency spike that lived inside one minute is invisible at 1-minute resolution, and percentiles cannot be reconstructed from sum/count/min/max because those aggregates don't carry the distribution. That is an accepted trade — you only downsample data old enough that nobody inspects it that closely, and full resolution is retained for the 48 hours where that detail matters. Common wrong answer to avoid: "Downsampling is lossless, just fewer points." It is lossy by design — you cannot recover a p99 or a sub-bucket spike from a rolled-up average — and pretending otherwise gets you burned when someone queries an old incident.

Q8. An ingester crashes. Do you lose data? Almost none. The head is in memory but every sample is also written to a write-ahead log on local SSD, which replays on restart, so at most the last few unflushed seconds are at risk — and with replication factor 3 and quorum writes, the other two replicas already hold those seconds, so the read path sees no gap. While the node is down the hash ring rebalances its shard onto peers, and when it returns the compactor's vertical merge unions any divergent blocks by (series_id, ts). The write path never stalls because success needs only 2 of 3 acks. Common wrong answer to avoid: "In-memory means a crash loses the recent data." That ignores the WAL and the replication factor; the whole point of both is that a single ingester death is a non-event.

Q9. How do you handle out-of-order and late samples? Samples generally must arrive in timestamp order per series because the chunk encoder appends forward, and delta-of-delta encoding assumes monotonic timestamps. Accept a bounded out-of-order window (a few minutes) by buffering, but reject anything older than the head's oldest open chunk, because absorbing it would mean rewriting an already-compressed or flushed chunk. A source with a badly skewed clock therefore loses its late samples rather than corrupting a sealed, immutable block. If genuinely late data matters (batch imports), route it through a separate ingestion path rather than the live head. Common wrong answer to avoid: "Just insert it wherever it belongs, sorted." Random-position inserts break the append-only compressed chunk model and would force decompress-rewrite on the hot path — the reason TSDBs restrict out-of-order writes in the first place.

Q10. How does a dashboard over 13 months of history stay fast? Three levers. The query automatically reads the coarsest resolution that answers it — 1-hour rollup blocks for a quarter-long view — so it scans thousands of points, not billions. The query frontend splits the range into per-day sub-queries and serves the immutable ones (anything older than the ~2-hour head) from a results cache, so a reloaded dashboard recomputes only its most recent, still-changing window. And postings intersection means even within each block only the handful of matched series are read, as contiguous compressed chunks, never a scan. A 30-day dashboard that would be 30 sub-queries costs one day's compute per refresh after the first load. Common wrong answer to avoid: "Add read replicas of the metric store." Replicas add throughput but not the resolution-tiering, range-splitting, and result-caching that actually make a year-long query cheap; you would just scan the same billions of points in parallel.

Q11. Two ruler replicas both fire the same alert. How do you avoid double-paging without risking a missed alert? Deliberately choose duplicate-and-dedupe over single-and-gap. Run rulers redundantly so a node loss never silently stops evaluating a rule, accept that both replicas emit the same firing alert, and let Alertmanager deduplicate by the alert's label fingerprint in its gossiping cluster so exactly one notification goes out. The asymmetry drives the choice: a duplicated alert is a cosmetic annoyance Alertmanager erases, while a missed alert during an incident is the failure the whole system exists to prevent. Common wrong answer to avoid: "Elect a single leader ruler so each rule evaluates once." A leader is a single point of silent failure for alert coverage — if it stalls, nobody gets paged and no one notices until the incident is already bad.

Q12. A host dies. Why might its dashboard still show a healthy value, and how do you fix it? Without intervention, the last scraped value has no successor, so a naive query carries it forward and the dead host's CPU appears frozen at its final reading — and an alert on "CPU > 90%" never clears because the value never changes. The fix is an explicit staleness marker: when a scrape fails or a pushed series stops, the ingester writes a stale marker into the series so queries return "no data" after that point rather than the stale last value. This makes "the host is gone" distinguishable from "the value happens to be steady," which is essential for both dashboards and alert resolution. Common wrong answer to avoid: "Just show the last known value." That is exactly the bug — a dead target looks alive, and threshold alerts stick firing or stuck resolved on frozen data.

Deeper follow-ups

• How would you support multi-tenancy so one team's cardinality spike or expensive query cannot degrade another team's ingestion or alerting?
• How would you handle exemplars (linking a metric spike to a specific trace) without inflating cardinality?
• Recording rules pre-compute expensive aggregates on a schedule — how do you keep them from double-counting or lagging, and where do their outputs get stored?
• How would you migrate the series-ID hash function or the chunk encoding without rewriting years of immutable blocks?
• Global querying across regions: do you replicate blocks, run a federated query layer, or accept per-region silos, and what does each cost in latency and consistency?
• How would you detect and attribute a slow cardinality creep (a few hundred new series an hour) that never trips the hard limit but doubles memory over a month?

How this round is scored

Interviewers use the metrics system to see whether you protect the write path and treat cardinality, not sample rate, as the scaling axis. The strong signal is naming early that a million samples a second is an index-churn problem, then building the compression, downsampling, and cardinality-limiting story around it — rather than reaching for a bigger database or fixating on ingest bandwidth that was never the bottleneck. Seniority shows in the tradeoff discussions: push versus pull decided by ephemerality and control, resolution versus cost resolved by tiered downsampling with an honest account of what detail is lost, and duplicate-and-dedupe versus single-and-gap on the alert path decided by which failure is worse. The failure-mode reasoning — the cardinality explosion, ingester crash recovery via WAL and replication, staleness markers, out-of-order rejection — separates candidates who have operated a metrics backend from those who have only drawn one. Doing the back-of-envelope math out loud (10M series ÷ 1M/s → a 10-second cadence → ~170 GB/day → ~1.3 TB downsampled) and using it to justify the retention tiers, not as decoration, is what pushes the answer from correct to senior.