02. Twitter/X Home Timeline — Low-Level Design¶

~18 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)

Data models¶

CREATE TABLE tweet (
    id          BIGINT PRIMARY KEY,     -- Snowflake (time-sortable)
    author_id   BIGINT NOT NULL,
    text        VARCHAR(280),
    media_id    UUID NULL,
    parent_id   BIGINT NULL,            -- reply target
    created_at  TIMESTAMP NOT NULL
);                                       -- sharded by id

CREATE TABLE retweet (                   -- a reference, NOT a copy
    id          BIGINT PRIMARY KEY,     -- Snowflake
    user_id     BIGINT NOT NULL,        -- who retweeted
    tweet_id    BIGINT NOT NULL,        -- the ORIGINAL tweet (flattened)
    created_at  TIMESTAMP NOT NULL
);

CREATE TABLE follow (follower_id BIGINT, followee_id BIGINT,
                     PRIMARY KEY(follower_id, followee_id));  -- + inverse index

-- Timeline cache (Redis):  tl:{user_id} = capped list of tweet_ids
-- Hot-object set (Redis):  hot_tweets = { tweet_id → velocity }, celeb posts

The retweet.tweet_id always points at the original tweet, even when someone retweets a retweet — the pointer is flattened at write time. This keeps hydration a single lookup and prevents retweet chains from ballooning.

Component internals¶

Fan-out service — dampened by hot-object detection¶

def on_tweet_event(evt):
    tid, author = evt.tweet_id, evt.author_id
    if hot_objects.is_hot(tid) or user_flags.is_celebrity(author):
        return                          # served via read-time merge, no fan-out
    for follower in graph.followers(author):
        timeline_cache.lpush_capped(f"tl:{follower}", tid, cap=800)

def on_retweet(evt):
    original = evt.tweet_id             # already flattened to the original
    if hot_objects.is_hot(original):
        return                          # viral: don't re-fan-out; it's a hot object
    for follower in graph.followers(evt.user_id):
        timeline_cache.lpush_capped(f"tl:{follower}", original, cap=800)

Both handlers share the guard: if the target tweet is hot, do nothing. Our breaking tweet crosses the velocity threshold within seconds; from that point its 1M retweets each hit is_hot → return, so the 415k-inserts/second storm collapses to a single hot object read by many timelines.

Hot-object detector — velocity threshold¶

def on_engagement(tweet_id):
    rate = velocity.increment(tweet_id)        # sliding-window retweets+likes / sec
    if rate > HOT_THRESHOLD and not hot_objects.is_hot(tweet_id):
        hot_objects.add(tweet_id)              # flip to pull path
        cache.warm(tweet_id)                   # pre-load body into hot-object cache

Detection must be early and cheap — a sliding-window counter per tweet, flipped the moment velocity crosses a low bar. Over-flagging is safe (it just moves a tweet to the cheap pull path); under-flagging is what causes the storm.

Timeline service — merge push list with hot objects¶

def build_timeline(viewer, cursor):
    pushed = timeline_cache.range(f"tl:{viewer}", cursor, limit=50)
    hot    = hot_objects.relevant_to(viewer)          # viral tweets + celebs followed
    merged = merge_by_id_time(pushed, hot)[:50]        # Snowflake ids sort by time
    return [hydrate(tid) for tid in merged]

def hydrate(tid):
    t = tweet_store.get(tid)                            # one lookup, even for retweets
    return { ...t, author: user_cache.get(t.author_id) }

Core algorithm — flattening retweet chains¶

retweet(user, target_id):
    t = tweet_store.get(target_id)
    original_id = t.original_id if t.is_retweet else target_id   # flatten
    insert retweet(user_id=user, tweet_id=original_id)
    emit Retweeted(tweet_id=original_id, user_id=user)

By always resolving to original_id, a retweet of a retweet of a retweet still points at one tweet. Hydration is therefore O(1) and the "retweet count" is a single counter on the original, not a traversal of a chain.

Sequence diagram — viral tweet dampening the storm¶

User      Tweet svc   Hot detector   Fan-out      Timeline cache   Reader   Timeline svc
 │ retweet  │             │             │              │            │           │
 ├──────────▶│ ref row    │             │              │            │           │
 │           ├─ Retweeted ─┼────────────▶│ is_hot? Y   │            │           │
 │           │             │             ├─ return (no fan-out)      │           │
 │           ├─ engagement ▶│ rate>thresh │              │            │           │
 │           │             ├─ add hot + warm cache       │            │           │
 │           │             │             │              │  GET /tl   │           │
 │           │             │             │              │            ├──────────▶│ merge push+hot
 │           │             │             │              │            │◀─ hot obj ─┤

Concurrency and edge cases¶

Detection race: many retweets arrive before the tweet is flagged hot. Accept a small burst of fan-out during the detection window; the sliding-window counter flips fast, and the pull path catches subsequent readers.
Retweet then delete original: hydration of a dangling reference returns a tombstone ("tweet unavailable"); references are not eagerly cleaned.
Ordering under clock skew: Snowflake worker/epoch bits bound cross-shard skew; timelines tolerate seconds of reordering.
Capped timeline overflow: beyond 800 ids, oldest drop; deep scroll falls back to a pull.
Duplicate in timeline: a tweet you receive both via push and via a hot-object merge is de-duped by id at merge time.
Live push vs durable timeline: the WebSocket push is best-effort; the durable timeline read is the source of truth, so a missed live event self-heals on next load.