Skip to content

01. Twitter/X Home Timeline — High-Level Design

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

Architecture

   post/retweet ─▶ ┌──────────────┐  event  ┌──────────────┐   ┌───────────────┐
                   │ Tweet svc    │────────▶│ Fan-out svc  │──▶│ Timeline cache│
                   └──────┬───────┘         └──────┬───────┘   │ (per-user ids)│
                          ▼                        │           └───────────────┘
                   ┌──────────────┐                │ (skip hot objects)
                   │ Tweet + Graph│◀───────────────┘
                   │ store        │        ┌───────────────┐
                   └──────────────┘        │ Hot-object    │ (viral tweets,
                                           │ cache         │  celebrity posts)
   timeline read ─▶ ┌──────────────┐       └───────────────┘
                    │ Timeline svc │─ merge(push ids + hot objects) ─▶ hydrate
                    └──────────────┘
   live ─▶ ┌──────────────────┐  push new tweets to open sockets
           │ Connection layer │◀── WebSocket ── clients
           │ (fan-out to conns)│
           └──────────────────┘

Components

Tweet service. Writes tweets and retweets (as references), emits TweetCreated/Retweeted.

Fan-out service. Pushes tweet ids into follower timelines, skipping hot objects and celebrity authors.

Timeline cache. Per-user capped list of tweet ids (the push path), in Redis.

Hot-object cache. Holds viral tweets and celebrity posts served via read-time merge instead of fan-out.

Tweet + Graph store. Source of truth for tweets, retweet edges, and the follow graph.

Timeline service. On read, merges the viewer's push list with hot objects (viral tweets + celebrities they follow), hydrates ids to tweet bodies.

Connection layer. Maintains millions of WebSocket connections and pushes new-tweet notifications to connected followers in real time.

Primary write path (tweet / retweet)

  1. POST /tweets (or /retweet) writes to the tweet store. A retweet writes a small reference row pointing at the original id.
  2. TweetCreated (or Retweeted) is emitted.
  3. Fan-out service decides: if the author is normal and the tweet is not a hot object, push the id into each follower's timeline cache. If the author is a celebrity, or the tweet has crossed the viral velocity threshold, skip — it will be pulled.
  4. The connection layer notifies online followers over their WebSocket so the tweet appears live.
  5. Posting returns as soon as the tweet row is written; fan-out and live push are asynchronous.

Primary read path (timeline)

  1. GET /timeline hits the timeline service.
  2. It reads the viewer's push list from the timeline cache.
  3. It fetches hot objects relevant to the viewer: viral tweets and recent tweets from celebrities they follow.
  4. It merges by tweet-id time order, paginates by cursor, and hydrates ids into bodies — resolving retweet references to the single original tweet.
  5. Returns the page.

Storage choices

  • Tweets: sharded, time-ordered ids. Immutable rows, sharded by tweet id or author; point lookups for hydration.
  • Retweets: reference rows, not copies — one original body, many pointers.
  • Follow graph: adjacency store, sharded by user, with an inverse index for follower enumeration.
  • Timeline cache: Redis capped lists per user.
  • Hot-object cache: Redis, small set of viral/celebrity tweets with high read fan-out.

Scaling

Timeline reads (180k/s) scale on the cache tier. The write challenge is the retweet fan-out storm: our breaking tweet generates ~415k timeline insertions/second if every retweet fans out. The dampening rule caps this — once the original tweet is flagged hot, retweets of it stop fanning out (it's already served via hot-object merge), so the storm collapses to "one hot object read by many timelines" instead of "hundreds of thousands of writes." Fan-out itself runs as a partitioned consumer group so normal-tweet fan-out parallelizes. The connection layer scales horizontally by sharding connections across nodes with a pub/sub backbone routing new-tweet events to the right nodes.

Operational signals

The healthy signal is timeline read p99 flat under a trending event. The first metric to degrade is fan-out queue depth, which spikes the instant a tweet starts going viral and before the hot-object flag catches it — the lag between "going viral" and "flagged hot" is the danger window. The misleading metric is total tweet write rate: it looks normal because the storm is fan-out insertions, not new tweets. The graph an operator opens first is fan-out insertions per originating tweet: a single tweet driving hundreds of thousands of insertions means the hot-object threshold fired too late — tighten it.

Failure modes and resilience

  • Viral tweet flagged too late. A window of runaway fan-out before the hot-object rule engages. Mitigation: detect velocity early (retweets/second crossing a low bar) and flag aggressively; over-flagging just shifts a tweet to the (cheap) pull path.
  • Fan-out backlog. Timelines lag. Mitigation: read-time pull of recent tweets from followed accounts as a backstop, so lag means slightly-stale, not missing.
  • Connection-layer node loss. Live updates drop for those users. Mitigation: clients reconnect and catch up via a normal timeline read; live push is best-effort on top of durable timelines.
  • Retweet-of-retweet chains. Must resolve to the single original, not a chain of copies. Mitigation: retweets reference the original id directly (flattened), never another retweet.
  • Hot-object cache miss on a viral tweet. Mitigation: request coalescing so one store read fills the cache for all concurrent timeline builds.

Where this shows up in production

  • Twitter/X — the canonical push/pull hybrid timeline with read-time merge for high-fan-out accounts; retweets stored as references.
  • Twitter's "fanout service" + Redis timelines — per-user timeline id lists in Redis are Twitter's documented design.
  • Kafka — the TweetCreated/Retweeted event backbone feeding partitioned fan-out workers.
  • WebSocket/gateway fleets (e.g. Twitter's stream) — millions of persistent connections for live delivery, sharded with a pub/sub router.
  • Facebook TAO / graph stores — follow-graph adjacency with follower enumeration for fan-out.
  • Snowflake IDs — Twitter invented Snowflake precisely for time-sortable tweet ids that make timeline merges cheap.
  • Redis pub/sub — routing new-tweet events to the connection node holding each follower's socket.