Skip to content

00. Design a Content Moderation System (ML in the Loop)

~20 min read · Level: intermediate · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A content moderation system decides, for every piece of user-generated content, whether it may stay up, must come down, or needs a human to look at it. This is the Trust & Safety machinery behind Facebook, YouTube, TikTok, and every marketplace that lets strangers upload text, images, and video. Someone posts a photo, a comment, or a listing; the platform has to catch child-safety material, terrorism, spam, fraud, and hate before it reaches an audience, without wrongly deleting the millions of ordinary posts that happen to trip a keyword. The same shape covers payment fraud, where the "content" is a transaction and the action is approve, decline, or send-to-manual-review.

What makes this a genuine system-design question is not the classifier. It is that a machine-learning model is fast but wrong a few percent of the time, and the two ways it can be wrong cost wildly different amounts. Letting harmful content through (a false negative) can put someone in danger; taking down a legitimate post (a false positive) angers a user and, at scale, becomes a censorship story. You cannot tune a single threshold to make both errors small, so you build a system that routes the confident cases to a machine and the ambiguous cases to people — and then you discover the people are the bottleneck.

Thread one scenario through the whole design: the platform ingests 1,000,000 uploads per hour, and every one is scored by an ML model. The model auto-acts on the cases it is confident about — roughly 95% auto-allowed, 2% auto-removed — and routes the uncertain 3%, about 30,000 items per hour, into a human review queue. Reviewers overturn some of those, and any user can appeal an automated decision, which can reverse it. That one flow — a million scored, most auto-acted, a fraction queued to humans, a slice appealed — tests every decision below.

Functional requirements

  • Score: run every submitted item through a moderation model (and a rules/hash pass) and produce a risk score plus policy labels (e.g. violence: 0.9, spam: 0.1).
  • Decide and act: map score and labels to an action — allow, remove, age-gate/limit reach, or route to human review — under policy-specific thresholds.
  • Human review queue: hold borderline items in a prioritized queue, let reviewers claim and decide them, and feed their verdicts back as labels.
  • Appeals: let a user contest an automated or human decision and have it re-reviewed, with the power to overturn and reinstate.
  • Audit trail: record who or what decided each item, on what evidence, immutably — regulators and internal review both demand it.

De-scoped for this round, and worth naming so the interviewer sees it as a choice: training the models themselves (we consume a served model, we don't design the training run), the upload/storage service for the raw media, and the full labeling-tool UX. We also skip proactive account-level abuse detection beyond per-item scoring. These matter, but they sit beside the core decision loop and don't change its architecture.

Non-functional requirements

The dominant constraint is sustaining scoring throughput while holding the false-negative rate under a hard ceiling — recall on the truly harmful categories is the number the whole architecture bends around. Everything else is negotiable against it.

  • Throughput: score 1M items/hour sustained, ~1,000/s at peak, with no unbounded backlog. Falling behind means unscored content is live.
  • Asymmetric error: a false negative on a high-severity category (child safety, terrorism) is far costlier than a false positive. The system must be able to fail closed for those categories and fail open for low-risk ones — one global setting is wrong.
  • Decision latency: two very different budgets. A cheap synchronous check (known-bad hash match, obvious spam) must return in well under 200 ms so the upload path can block instantly. Deep ML scoring runs asynchronously and may take seconds; borderline items may wait minutes in the human queue. There is no single latency SLA.
  • Auditability & consistency: each item's decision is a durable, append-only record. A producer that polls for an item's state must read its own latest decision (read-your-writes); analytics on aggregate rates can lag.
  • Availability: the scoring path must degrade safely, never silently. If the model is down, the fail-open/fail-closed policy decides what happens — the system does not just drop items on the floor.

Scale estimation

Start from the threaded numbers. 1,000,000 uploads/hour is 1,000,000 / 3,600 ≈ 278 items/second on average. Trust & Safety traffic is spiky — evening peaks, viral events, coordinated abuse campaigns — so apply a ~3.5× peak factor and design the scoring path for ~1,000 items/second at peak.

Human queue. The 3% borderline share is 0.03 × 1,000,000 = 30,000 items/hour, or ~8.3 items/second, entering the review queue. Sizing the reviewer pool is Little's Law, not guesswork: if the average handling time (AHT) per item is 60 seconds, one reviewer clears 3,600 / 60 = 60 items/hour, so draining 30,000/hour needs 30,000 / 60 = 500 reviewers working concurrently. Provision ~20% headroom and you staff ~600. This number is the hidden cost of every ML threshold decision — tighten the model to catch more harm and the borderline share rises, and the reviewer count rises with it.

Auto-actions. 95% auto-allow is 950,000/hour; 2% auto-remove is 20,000/hour. Appeals run at maybe 5% of removals, ~1,000 appeals/hour, of which perhaps 15% overturn — ~150 wrongful removals corrected/hour, which is also our running measurement of auto-block precision.

Compute. If one GPU inference replica scores ~100 items/second (batched), covering the 1,000/s peak needs 1,000 / 100 = 10 replicas, plus headroom for redundancy and model rollout — call it ~14. Average load needs only ~3, so autoscaling between 3 and 14 tracks the diurnal curve.

Storage. We store a decision record, not the raw media (that lives in the upload service). A decision row — content id, scores, labels, action, actor, timestamps — is ~1 KB. At 1M/hour that is 1,000,000 × 1 KB = 1 GB/hour ≈ 24 GB/day ≈ 8.7 TB/year of decision and audit data, plus feature vectors/embeddings if we cache them (another ~1 KB each). That is a modest warehouse, not a big-data problem; the interesting scale is throughput and human capacity, not bytes.

API sketch

POST /v1/content/submit
  body: { "content_id": "...", "type": "image|text|video", "uri": "...",
          "author_id": "...", "context": {...} }
  200:  { "decision": "allow" | "block" | "pending", "risk_score": 0.07,
          "labels": {"spam": 0.05, "violence": 0.02}, "decision_id": "..." }

GET  /v1/content/{content_id}/decision
  200:  { "state": "allowed|removed|pending_review|appealed", "actor": "model|reviewer",
          "score": 0.07, "history": [...] }

POST /v1/review/claim            # reviewer pulls the next item off the queue
  200:  { "review_item_id": "...", "content_uri": "...", "model_labels": {...}, "lease_expires_at": ... }

POST /v1/review/{review_item_id}/decide
  body: { "verdict": "keep|remove|escalate", "policy_labels": [...], "verdict_token": "..." }
  200:  { "state": "resolved" }

POST /v1/appeals
  body: { "content_id": "...", "author_id": "...", "reason": "..." }
  200:  { "appeal_id": "...", "state": "queued" }

Solutioning

Begin with the asymmetric-error fact and the architecture follows. Because one model threshold cannot make both false negatives and false positives small, you do not pick a threshold — you pick two and cut the score line into three bands. Everything below a low threshold is confidently safe and auto-allowed; everything above a high threshold is confidently harmful and auto-removed; the band in the middle is the ambiguous 3% that goes to humans. The whole system is a triage funnel that spends cheap machine capacity on the easy 97% so it can spend expensive human capacity only on the hard 3%. The reframing to carry into the room: a borderline post is not a model-accuracy problem; it is a queue-capacity problem — you will never model your way out of the ambiguous middle, so you design the queue that drains it.

The second decision is synchronous block versus asynchronous review, and the answer is tiered rather than either/or. Some checks are cheap and deterministic — a perceptual-hash match against a known-bad database (PhotoDNA-style) has effectively zero false positives on a hit, so you run it synchronously at upload and block instantly, in under 200 ms. Deep ML scoring is too slow for the upload path at 1,000/s, so it runs asynchronously: the item is accepted, scored within seconds, and acted on after the fact. For low-risk content types you publish optimistically and remove if the async score comes back bad; for high-severity types you hold the item in pending until scored — you fail closed. The single sync/async knob is per-category, driven by the cost of being wrong for that category.

The third decision is the ML-plus-rules blend, and each leg exists because the other cannot do its job. Rules and hash matches give you precision on known bad content — an exact hash hit is certain, and a curated rule for a specific banned phrase won't wrongly flag novel posts. ML models give you recall on novel bad content the rules have never seen, at the cost of probabilistic errors. Humans give you judgment on the genuinely ambiguous cases neither can settle. So the pipeline is deterministic-rules-first (cheap, certain, catches the known), ML-second (broad, probabilistic, catches the novel), human-third (scarce, final, settles the ambiguous). Stacking them this way means the confident layers shed load off the expensive ones: the hash pass removes known-bad before it ever costs a GPU cycle, and the model auto-acts on 97% before it ever costs a reviewer minute.

Two smaller decisions finish the shape. Appeals are a first-class reverse path, not a support ticket — an overturned decision must reinstate the content and correct the stored label so the mistake becomes training signal rather than repeating. And the human verdicts are the freshest, highest-quality labels in the building, so the review queue doubles as a labeling pipeline feeding the next model. The result is a system whose fast path is a synchronous hash/rule check, whose main path is asynchronous ML scoring into a three-band decision engine, whose hard cases drain through a prioritized human queue sized by Little's Law, and whose mistakes flow back through appeals into the training set. The following files take each of these down to components (HLD) and then to schemas, algorithms, and edge cases (LLD).