02. Content Moderation — Low-Level Design¶
~20 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)
The HLD named the boxes. This file opens the three that carry the design's weight — the decision engine that bands scores into actions, the review queue that leases work to humans, and the appeals path that reverses mistakes — and pins down the data, the algorithms, and the concurrency corners where a moderation system actually breaks.
Data models¶
The system of record is a current-state row per content item plus an append-only event log. In a relational store:
CREATE TABLE content_decision (
content_id VARCHAR(64) PRIMARY KEY, -- from the upload service; the shard key
state SMALLINT NOT NULL, -- 0=pending 1=allowed 2=removed 3=limited 4=in_review 5=appealed
action_actor SMALLINT NOT NULL, -- 0=rules 1=model 2=reviewer 3=appeal
risk_score REAL NULL, -- calibrated top-label probability
labels JSONB NOT NULL, -- {"violence":0.9,"spam":0.1,...}
policy_version VARCHAR(32) NOT NULL, -- which threshold set decided this
model_version VARCHAR(32) NULL, -- which model produced the score
decided_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
-- Append-only audit: never updated, only inserted. Regulators read this.
CREATE TABLE decision_event (
event_id BIGINT PRIMARY KEY,
content_id VARCHAR(64) NOT NULL,
seq INT NOT NULL, -- monotonic per content_id
actor SMALLINT NOT NULL,
action SMALLINT NOT NULL,
evidence JSONB NOT NULL, -- scores, rule hits, reviewer id, appeal id
created_at TIMESTAMP NOT NULL,
UNIQUE (content_id, seq)
);
Three deliberate choices. First, content_decision holds only the current state for fast point-lookup, while decision_event holds the full immutable history — mutating current state is cheap, and the append-only log is what you hand a regulator, so they are separated on purpose. Second, policy_version and model_version are stored on every decision because "why was this removed" is unanswerable without knowing which thresholds and which model decided it — this is what makes a bad model deploy diagnosable after the fact. Third, labels is stored per-category, not a single score, because the fail-closed rule is per category — a 0.9 on violence and a 0.9 on spam route very differently.
The review queue is its own store, optimized for priority dequeue and leasing:
CREATE TABLE review_item (
review_item_id VARCHAR(64) PRIMARY KEY,
content_id VARCHAR(64) NOT NULL,
priority INT NOT NULL, -- severity × exposure, higher = sooner
labels JSONB NOT NULL, -- model's evidence, shown to reviewer
state SMALLINT NOT NULL, -- 0=queued 1=leased 2=resolved
leased_by VARCHAR(64) NULL, -- reviewer id holding the lease
lease_expires TIMESTAMP NULL, -- visibility timeout
enqueued_at TIMESTAMP NOT NULL
);
CREATE INDEX idx_dequeue ON review_item (state, priority DESC, enqueued_at)
WHERE state = 0; -- partial index: only the queue front
The partial index on state = 0 is the whole trick for the priority dequeue — the "next item" query scans only unclaimed items, ordered by priority then age, so a queue of millions doesn't slow the pull. Appeals reuse this table with a higher priority band and a source = appeal flag so they jump ahead of routine review.
CREATE TABLE appeal (
appeal_id VARCHAR(64) PRIMARY KEY,
content_id VARCHAR(64) NOT NULL,
author_id VARCHAR(64) NOT NULL,
state SMALLINT NOT NULL, -- 0=queued 1=in_review 2=upheld 3=overturned
original_actor SMALLINT NOT NULL, -- what made the contested decision
created_at TIMESTAMP NOT NULL,
UNIQUE (content_id, author_id) -- one open appeal per item per user
);
Component internals¶
Component 1 — Decision Engine (score → action banding)¶
Responsibility: turn calibrated label scores and rule hits into an action, under per-category, versioned policy, with an explicit fail direction.
class DecisionEngine:
def decide(content, labels: dict, rule_hits: list, ctx) -> Decision
def _band(category, score) -> Band # ALLOW | REVIEW | BLOCK
def _fail_direction(category) -> Action # PENDING (closed) | ALLOW (open)
class PolicyConfig: # versioned, hot-reloadable, no model redeploy
thresholds: dict[category, (low, high)] # e.g. violence:(0.20,0.95)
severity: dict[category, int]
fail_closed: set[category] # child_safety, terror, ...
The engine evaluates the most severe triggered category, not an average. A rule hit is treated as a hard signal (score 1.0 for its category). For each category it consults that category's (low, high) thresholds: below low is confidently safe, above high is confidently harmful, between is the ambiguous band that routes to review. The overall action is the strictest across categories — one violence: 0.96 removes the item even if everything else is 0.01. When a score is missing (model down) the engine uses _fail_direction: a fail_closed category holds the item pending; everything else stays allow and is rescored later.
Component 2 — Review Queue Manager (priority, lease, redrive)¶
Responsibility: hand each queued item to exactly one reviewer, in priority order, and recover items whose reviewer never finishes.
class ReviewQueue:
def enqueue(content_id, priority, labels) -> review_item_id
def claim(reviewer_id, lease_secs=180) -> ReviewItem | None # atomic dequeue+lease
def submit(review_item_id, verdict, verdict_token) -> None # idempotent
def redrive() -> int # sweep expired leases
claim is the concurrency-critical operation: it must select the highest-priority queued item and lease it to one reviewer atomically, so two reviewers never get the same item. In SQL that is a single UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED); in Redis it is an atomic ZPOPMAX plus a lease key. SKIP LOCKED is what lets 500 reviewers pull concurrently without blocking each other — each grabs the next unlocked front-of-queue row. redrive runs on a timer and returns any item whose lease_expires has passed back to queued, so a reviewer who closes their laptop mid-item doesn't strand it forever.
Component 3 — Priority function (severity × exposure)¶
Responsibility: decide which borderline item a scarce reviewer sees first.
def priority(item) -> int:
severity = SEVERITY[item.top_category] # child_safety=100, violence=60, spam=5
exposure = log10(1 + author_follower_count) # reach if it stays up
age_boost = minutes_in_queue(item) * 0.5 # anti-starvation
return severity * 10 + exposure + age_boost
Exposure is why a borderline post from an account with 10M followers is reviewed before an identical one from an account with 10 followers — the cost of leaving it up for five extra minutes is a thousand times larger. The age_boost prevents low-priority items from starving forever when the queue is hot.
Core algorithm — three-band decision, walked with the scenario's numbers¶
Take the threaded hour: 1,000,000 items scored. Each produces a calibrated top-label score in [0, 1]. Assume the policy for the dominant category sets low = 0.20 and high = 0.95. The engine bands every item:
- Score < 0.20 → auto-ALLOW. For a well-calibrated model on this traffic, ~95% of items land here:
0.95 × 1,000,000 = 950,000 items/hourpublished with no human cost. - Score > 0.95 → auto-BLOCK. ~2% land here:
0.02 × 1,000,000 = 20,000 items/hourremoved automatically, each written to the audit log with its score and model version. - 0.20 ≤ score ≤ 0.95 → REVIEW. The ambiguous ~3%:
0.03 × 1,000,000 = 30,000 items/hour ≈ 8.3/senqueued, each assignedpriority = severity × 10 + exposure + age_boost.
Now size the drain with Little's Law. To keep the queue bounded at λ = 8.3/s with AHT = 60 s, you need L = λ × W... reading it the other way, concurrent reviewers = λ × AHT = 8.3 × 60 ≈ 500. To hold the oldest-item age under a 5-minute SLA, the number that matters is service rate ≥ arrival rate with headroom: 500 reviewers give exactly 500 / 60 ≈ 8.3/s service, which is break-even and therefore fragile, so you staff ~600 (10/s service) to keep wait time comfortably under SLA.
Watch what a precision-versus-recall move does to these numbers. Suppose analysts decide the auto-block band is too aggressive — it's catching legitimate posts, and appeals confirm a false-positive problem. They raise high from 0.95 to 0.98 to remove only the most certain cases. Auto-blocks fall from 2% toward ~1.2% (~12,000/hour, 8,000 fewer wrongful-ish removals), but those 8,000 items don't vanish — they slide into the review band. Borderline arrivals rise from 8.3/s to about 10.5/s, and required reviewers jump from 500 to 10.5 × 60 ≈ 630. The precision improvement was paid for in reviewer headcount. That coupling — every threshold nudge is a staffing change — is the algorithm's real lesson.
Sequence diagram — an item from submit to human verdict¶
Producer Gateway Rules Scoring DecisionEng ReviewQueue Reviewer DecisionStore
│ submit │ │ │ │ │ │ │
├─────────▶│ dedup │ │ │ │ │ │
│ ├─ hash?───▶│ │ │ │ │ │
│ │◀─ no hit ─┤ │ │ │ │ │
│◀ pending ┤ enqueue async ───────▶│ │ │ │ │
│ │ │ score ──▶│ │ │ │ │
│ │ │ ├─ labels ▶│ │ │ │
│ │ │ │ ├─ band=REVIEW (0.20..0.95)│ │
│ │ │ │ ├─ enqueue(pri) ─────────▶│ │
│ │ │ │ ├─ write pending ─────────┼───────────▶│
│ │ │ │ │ │ claim ◀───┤ │
│ │ │ │ │ ├─ lease ───▶│ │
│ │ │ │ │ │ ├─ verdict ─▶│ (remove)
│ │ │ │ │ │◀ submit ───┤ │
│ │ │ │ │ ├─ resolve ──┼───────────▶│ + label
│ GET decision ◀───────────────────────────────────────────────────────────────────┤ removed
The producer's submit returns pending the moment the synchronous hash check clears; everything after the enqueue is asynchronous. The item is scored, banded into REVIEW, written as pending and enqueued, then leased to one reviewer whose verdict both resolves the decision store and emits a training label. A later GET decision reads the resolved state.
Concurrency and edge cases¶
- Double-claim race. Two reviewers calling
claimat the same instant must not get the same item. The atomic dequeue-and-lease (SELECT ... FOR UPDATE SKIP LOCKEDorZPOPMAX+ lease key) makes the claim indivisible, so each reviewer takes a distinct front-of-queue item and neither blocks the other. - Stranded lease. A reviewer claims an item, then crashes or walks away. The
lease_expiresvisibility timeout plus theredrivesweep return it toqueuedafter 180 s, so no item is lost to an abandoned lease — at the cost that a genuinely slow review may be redriven and shown to a second reviewer (acceptable; the verdict is idempotent). - Idempotent verdict. A reviewer's
submitcarries averdict_token; a network retry re-sends the same token, and the queue manager treats a second submit with the same token as a no-op. This stops a double-click from removing content twice or racing with a redrive that reassigned the item. - Late score after human decision. The async model score can arrive after a human already resolved the item (redrive reassigned it, or an appeal reopened it). Precedence is explicit and stored in
action_actor: a human verdict outranks a model score, and an appeal outranks both. A late model score for an item in a terminal human state is recorded in the audit log as evidence but does not change the action. - Idempotent submission. The upload service may retry
submitafter a timeout. The gateway dedups oncontent_id, returning the existing decision rather than scoring the item twice — critical because double-scoring could enqueue the same item to two reviewers. - Appeal versus in-flight rescore. An appeal reopens a resolved decision while a periodic rescore might also be re-evaluating it. The
content_decision.statemachine gates transitions — an item inappealedstate ignores model-driven transitions until the appeal resolves — so the two paths can't both mutate current state and clobber each other. - Read-your-writes for the producer. After an action executes, the decision store write and the audit append happen before (or in the same transaction as) the state flip, so a producer polling
GET decisionimmediately after never sees a stalependingfor an item that has actually been acted on. - Fail-direction under model outage. If the score is absent, the engine does not guess. A
fail_closedcategory holdspending(nothing dangerous publishes unscored) and everything else staysallowpending a rescore — the missing-data path is a policy decision, not an exception that drops the item.