01. Content Moderation — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the triage funnel from the overview into concrete boxes and the flows between them. Read the architecture top to bottom, follow one item through scoring and into the human queue, then look at what happens when the model or the reviewers can't keep up.

Architecture¶

   producer (upload service) ──▶ ┌──────────────────┐
                                 │ Ingestion Gateway │  dedup, accept, sync fast-path
                                 └─────────┬─────────┘
                                           │
                        ┌──────────────────┴───────────────────┐
                        ▼ (synchronous, <200ms)                 │ (async)
                 ┌─────────────┐                                ▼
                 │ Hash / Rules │  known-bad exact         ┌──────────────┐
                 │   Engine     │  match → block now        │ Scoring Queue │
                 └──────┬───────┘                           │  (durable log)│
                        │ no hit                            └──────┬───────┘
                        └───────────────────────────▶              ▼
                                                          ┌──────────────────┐
                                                          │  ML Scoring Svc    │◀── Feature
                                                          │ (GPU inference,    │    Store
                                                          │  calibrated score) │
                                                          └─────────┬─────────┘
                                                                    ▼
                                                          ┌──────────────────┐
                                                          │  Decision Engine   │  3-band policy:
                                                          │ (policy+thresholds)│  allow / block / review
                                                          └───┬───────┬───────┘
                                          auto-act ◀──────────┘       └────────▶ borderline
                                                │                              │
                                    ┌───────────▼──────┐          ┌────────────▼─────────┐
                                    │  Action Executor  │          │  Human Review Queue   │
                                    │ allow/remove/limit│          │ (priority, leased)    │
                                    └───────────┬──────┘          └────────────┬─────────┘
                                                │                              ▼
                                                │                     ┌──────────────────┐
                                                │                     │ Reviewer Console  │
                                                │                     └────────┬─────────┘
                                                ▼                              │ verdict
                                    ┌──────────────────┐◀─────────────────────┘
                                    │  Decision Store   │───▶ Label / Feedback ──▶ (retraining)
                                    │  + Audit Log      │      Store
                                    └────────┬─────────┘
                                             ▲
                                    ┌────────┴─────────┐
                                    │  Appeals Service │◀── user appeal (reopens a decision)
                                    └──────────────────┘

Read it as a funnel that widens only where it must. Every upload enters the ingestion gateway, which pays one cheap synchronous check — a hash/rules lookup that can block a known-bad item in under 200 ms without ever touching a GPU. Everything that survives that check flows asynchronously through a durable scoring queue into the ML scoring service, which produces a calibrated risk score. The decision engine cuts that score into three bands: confident-safe and confident-harmful items are handed to the action executor to auto-act, while the ambiguous middle is routed to the human review queue. Reviewer verdicts and automated actions both land in the decision store, which is the append-only system of record, and also feed the label store that trains the next model. The appeals service is the reverse arrow — it reopens a decision in the store and can overturn it.

Components¶

Ingestion Gateway. Accepts submissions from the upload service, deduplicates by content_id so a retried upload isn't scored twice, and runs the synchronous fast path. It returns allow, block, or pending immediately. It exists to make the common cheap decisions inline and get everything else onto the async rail fast, so the producer is never blocked on a GPU.

Hash / Rules Engine. The deterministic leg. It matches content against perceptual-hash databases of known-bad media (child-safety and terror hash sets) and against curated regex/keyword and metadata rules. A hash hit is certain, so it blocks synchronously with effectively zero false positives. It exists to shed known-bad load before it costs a model inference.

Scoring Queue. A durable, replayable log (Kafka or equivalent) between acceptance and scoring. It absorbs bursts — the gap between 278/s average and 1,000/s peak — and decouples ingestion availability from model availability. If scoring lags, items wait here durably rather than being lost.

ML Scoring Service. The GPU inference tier. It enriches each item with features (author history, embeddings, prior signals) from the feature store, runs the batched model, and emits a calibrated probability per policy label. Calibration matters: the decision engine's thresholds are only meaningful if a score of 0.9 truly means ~90% likely violating. It autoscales between ~3 and ~14 replicas with load.

Decision Engine. The policy brain. It maps scores and rule hits to an action using per-category, two-threshold banding, and encodes the fail-open/fail-closed choice per content type. It is deliberately separate from the model so policy can change — raise a threshold, add a category — without redeploying the model, and vice versa.

Action Executor. Applies the decision to the platform: allow (publish/keep), remove, or limit (age-gate, demote reach, restrict sharing). It is the one component with side effects on user-visible state, so it is idempotent and writes what it did to the audit log.

Human Review Queue. A prioritized, leased work queue holding the borderline 3%. It orders by severity and exposure, hands items to reviewers under a lease (visibility timeout), and redrives items whose lease expires. This is the scarce-capacity component the whole design protects.

Reviewer Console + Verdict path. The UI where a reviewer sees the item, the model's labels, and context, and records a verdict. Verdicts are final actions (they call the action executor) and simultaneously high-quality labels (they flow to the label store).

Decision Store + Audit Log. The durable system of record: current state per content item plus an append-only history of every automated and human action, with actor and evidence. Point-lookup by content_id for the producer's polling, append-only for audit and regulators.

Appeals Service. Reopens a resolved decision on user request, routes it back through review (often to a different, senior reviewer), and on overturn drives the action executor to reinstate content and writes a corrected label.

Primary write path (an item is scored and acted on)¶

POST /v1/content/submit reaches the ingestion gateway, which dedups on content_id (a retry returns the existing decision).
The gateway runs the hash/rules check synchronously. A hash hit returns block in under 200 ms and writes the decision — done, no model needed.
Otherwise the gateway writes the item to the scoring queue and returns pending (or allow optimistically, for low-risk types that publish-then-check).
The ML scoring service consumes the item, enriches it from the feature store, runs batched inference, and emits calibrated label scores.
The decision engine applies the policy bands. Below the low threshold → allow; above the high threshold → remove; in between → route to the review queue. High-severity categories that are unscored or on the boundary fail closed to pending/hold.
Auto-actions go straight to the action executor; the decision and its evidence are written to the decision store + audit log.
Borderline items are enqueued to the human review queue with a priority; a reviewer later claims and decides them, which calls the action executor and writes both a decision record and a training label.

Primary read path (state lookups and reviewer pulls)¶

There are two distinct reads. The producer calls GET /v1/content/{id}/decision to learn an item's current state; this is a point lookup on the decision store and must be read-your-writes so a just-acted item reports its real state, not a stale pending. The reviewer calls POST /v1/review/claim to pull the next item; this is a priority dequeue that atomically leases the highest-priority unclaimed item to exactly one reviewer for a bounded time. Both reads are latency-sensitive in different ways: the producer wants freshness, the reviewer wants the right next item — highest severity and exposure first — so the queue read is a priority operation, not FIFO.

Storage choices¶

Decision store + audit: partitioned relational or wide-column, append-only history. Access is point-lookup by content_id plus an immutable event log per item. A partitioned SQL store or a wide-column store (Cassandra/Spanner) fits: clean shard key on content_id, strong per-item consistency for read-your-writes, and an append-only child table for audit that regulators can be handed.
Scoring queue: durable log (Kafka). Chosen for replayability and burst absorption. If scoring falls behind, unscored items sit here durably and replay on recovery — the queue is the buffer that keeps a slow model from becoming lost content.
Review queue: priority queue with leases (Redis sorted sets or a DB-backed queue), not Kafka. Kafka's strict-partition-order model fights priority reordering and per-item leasing. The review queue needs to serve the highest-priority item next, lease it, and redrive on timeout — a sorted-set/priority structure with visibility timeouts, not a log.
Feature store: low-latency KV for online features + warehouse for offline. The scoring service needs author-history and prior-signal features in single-digit milliseconds, so an online KV (Redis/DynamoDB) fronts a warehouse used for training.
Label / feedback store: append-optimized warehouse. Verdicts and overturned appeals are append-heavy and queried by aggregation for training and metrics — a columnar warehouse (BigQuery/ClickHouse), kept separate so training queries never touch the live decision path.

Scaling¶

Scoring path. Throughput scales with GPU replicas behind the scoring queue. At 1,000/s peak and ~100/s per replica you run ~10–14 replicas; a model that's 2× heavier per item doubles that to ~20–28, which is why inference cost is a first-class budget line. The queue smooths bursts, so replicas track sustained load, not instantaneous spikes, and autoscale down to ~3 overnight.

Human queue — where the real scaling lives. The queue drains only as fast as reviewers work. At λ = 8.3 items/second arriving and 60 s AHT, Little's Law fixes the staff at λ × AHT = 8.3 × 60 ≈ 500 concurrent reviewers to hold steady. Now watch the coupling: if Trust & Safety tightens the ML thresholds to raise recall on a new abuse type, the borderline band grows from 3% to 5%, arrivals jump to 50,000/hour ≈ 13.9/s, and required staff climbs from 500 to 13.9 × 60 ≈ 830. You cannot scale reviewers like GPUs — hiring and training take weeks — so threshold changes are staffed changes, planned ahead, not shipped on a Friday.

Ingestion and stores. The gateway is stateless and scales horizontally behind a load balancer. The decision store shards on content_id, which distributes uniformly, so no shard hotspots from key skew. The synchronous hash check is an in-memory set / bloom filter fronting the hash DB, so it adds well under 10 ms even at 1,000/s.

Operational signals¶

The healthy signal is a flat, bounded review-queue depth with the oldest-unclaimed-item age sitting under its SLA (say 5 minutes) — a queue that stays shallow means arrivals and reviewer capacity are matched. The first metric to degrade under trouble is that oldest-item age: when arrivals exceed service rate, depth and wait time climb linearly long before anything errors, so queue age is the leading indicator of a staffing or threshold problem. The misleading metric is aggregate model accuracy or AUC — it can look healthy at 0.98 while recall on one newly-emerging abuse category has quietly collapsed, because that category is a rounding error in the aggregate; watch per-category recall and the auto-action mix, not the global number. The graph an experienced operator opens first during an incident is the arrival-rate-versus-service-rate overlay on the review queue (and the scoring backlog/lag on the async path): if the arrival line has crossed above the service line, the backlog is growing and every minute makes it worse, which tells them immediately whether to shed load, shift thresholds, or surge staff.

Failure modes and resilience¶

ML scoring outage or backlog. Items pile in the scoring queue unscored. This is the threaded scenario's dangerous case: at 278/s arriving and scoring throughput dropped to, say, 150/s, the backlog grows at 128/s ≈ 7,680/minute, so a ten-minute model outage leaves ~77,000 items unscored and live. Resilience is the fail-open/fail-closed policy per category: high-severity content holds in pending (fail closed) so nothing dangerous publishes unscored, while low-risk content publishes optimistically (fail open) and is rescored on recovery. The queue is durable, so nothing is lost — only delayed.
Human queue overload. Reviewers fall below the needed 500 — say a shift is short and only 400 are online, giving 400 × (1/60) ≈ 6.7/s service against 8.3/s arrival. The queue grows at 1.6/s ≈ 96/minute, so after one hour the backlog is ~5,760 items and the oldest-item age blows the SLA. Levers: shed by priority (serve highest severity/exposure first and let low-risk borderline items auto-resolve with conservative thresholds), temporarily widen the auto-allow band, and surge on-call reviewers. The design goal is that overload degrades the least risky items first, never the child-safety ones.
Bad model deploy. A regression tanks precision, false positives spike, legitimate content is removed, and appeals flood. Resilience: shadow-score new models against the live one, canary to 1% of traffic, and auto-rollback when canary precision drops below a floor. Appeals volume is itself a rollback signal.
Poisoned feedback loop. If overturned appeals and reviewer verdicts feed training uncritically, coordinated bad actors can teach the model to allow their content. Mitigation: weight and audit label sources, require senior review on appeal overturns for high-severity categories, and hold out a trusted golden set to detect drift.
Audit/consistency gap. If an action executes but its audit record is lost, the platform can't defend a decision to a regulator. Mitigation: write the decision record in the same transaction as (or before) the user-visible action, and treat the append-only log as the source of truth the executor reconciles against.

Where this shows up in production¶

Meta / Facebook — runs a classifier cascade that auto-acts on high-confidence content and routes the uncertain middle to a global human review workforce, failing closed on child-safety categories exactly as described here.
Microsoft PhotoDNA — the perceptual-hash matcher for known child-safety media that platforms run synchronously at upload; the deterministic, zero-false-positive-on-hit leg of our rules engine.
YouTube Content ID — hash/fingerprint matching against a reference database, showing how the exact-match leg scales to millions of uploads before any ML is involved.
Stripe Radar — inline ML fraud scoring that maps a risk score to allow / block / manual-review bands, the payments version of our three-band decision engine with a sync-block budget.
TikTok — holds some content categories in pre-publish review while publishing others optimistically, the per-category sync-vs-async tiering decision made explicit.
Google Perspective API / Jigsaw — a served toxicity-scoring model consumed by other products, illustrating the "we consume a served model, not the training run" boundary.
Reddit / Discord — human moderator queues with prioritization and escalation, the scarce-capacity review tier that ML triage feeds.
Airbnb / marketplace trust teams — risk-score listings and users, then route the ambiguous ones to manual review, coupling model thresholds to reviewer staffing the way our Little's Law math does.