01. Collaborative Editor — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, then follow an edit through the write path and a document open through the read path, then look at what happens when pieces fail — including what happens when the one paragraph gets 50 editors at once.

Architecture¶

   editor A          editor B          editor C ... (50 clients)
   (local doc +      (local doc +      (local doc +
    pending queue)    pending queue)    pending queue)
        │  ▲              │  ▲              │  ▲
        │  │ ops / acks / presence over WebSocket
        ▼  │              ▼  │              ▼  │
   ┌──────────────────────────────────────────────────┐
   │              WebSocket Gateway tier                │  holds connections,
   │           (stateless, ~50k conns each)             │  relays to owner
   └───────────────┬───────────────────┬────────────────┘
                   │ route by doc_id    │ presence
                   ▼                    ▼
        ┌────────────────────┐   ┌───────────────┐
        │  Doc Session Server │   │  Presence /   │
        │  (ONE per doc_id):  │   │  Pub-Sub      │
        │  serialize → assign │   │ (Redis, TTL,  │
        │  rev → transform →  │   │  ephemeral)   │
        │  broadcast          │   └───────────────┘
        └────┬──────────┬─────┘
             │ append    │ periodic
             ▼           ▼
      ┌────────────┐  ┌───────────────┐
      │  Op-Log     │  │  Snapshot      │
      │ (wide-col,  │  │  store         │
      │ doc_id/rev, │  │ (blob/doc DB,  │
      │ append-only)│  │  doc_id+rev)   │
      └────────────┘  └───────────────┘

Read it as three lanes. Down the middle is the edit lane: clients hold a local copy plus a queue of edits they've made but the server hasn't acknowledged; they speak over persistent WebSockets to a stateless gateway tier whose only job is to hold connections and route each document's traffic to the one server that owns it. That owner is the document session server — exactly one authoritative process per active doc_id — which serializes every incoming edit into a single total order, assigns it a revision number, transforms it against anything it missed, appends it durably, and broadcasts it back out. To the right is the presence lane, a fast ephemeral pub-sub for cursors and "who's here" that is deliberately kept off the durable path. At the bottom is the persistence lane: an append-only op-log that is the system of record and a periodic snapshot that makes reopening a document cheap.

Components¶

Client (editor + local model + pending queue). More than a text box. It holds the authoritative-for-now local document, applies every keystroke optimistically so typing feels instant, and keeps a pending queue of ops it has sent but not yet had acknowledged. When a peer's op arrives, the client transforms it against its own pending ops before applying, so the two never corrupt each other. This is half the OT algorithm; the server is the other half.

WebSocket gateway tier. Stateless connection holders. Each box terminates tens of thousands of long-lived sockets, and its job is routing: send every edit for a document to that document's session server (via consistent hashing on doc_id) and relay acks and broadcasts back. Because gateways hold no document state, one dying just drops connections that clients immediately re-establish elsewhere. This tier is what lets connection count and document-authority scale independently.

Document session server. The heart of the system and the one stateful, non-shardable piece. For each active document, exactly one session server owns it, holds the current materialized text and current revision in memory, and is the single serialization point that turns concurrent edits into one agreed order. It runs the OT transform, assigns monotonic revisions, appends to the op-log, and fans out. Ownership is guarded by a lease (below) so that two servers can never both claim one document.

Presence / pub-sub. An in-memory, TTL'd store (Redis or equivalent) for cursor positions, selections, and collaborator lists. Presence is fire-and-forget and self-expiring: a client heartbeats its cursor every couple of seconds, and if it goes silent its entry ages out. Nothing here is durable, and nothing here is allowed to block an edit.

Op-log store. The durable system of record: an append-only log partitioned by doc_id and ordered by rev. Every acknowledged edit is written here before the ack is sent, so an acknowledged edit is never lost. Its only access patterns are append-one and read-an-ordered-range-by-doc — exactly what a wide-column/LSM store does best.

Snapshot store. Periodically materialized full-document text keyed by doc_id and the rev it represents. It exists purely to bound cold-open cost: instead of replaying a million ops, a client loads the latest snapshot and replays only the short tail since.

Primary write path (a keystroke becomes an agreed edit)¶

A user types. The client applies the edit locally immediately (0 ms feedback), appends it to the pending queue tagged with base_rev — the latest server revision the client has seen — and a per-client client_seq.
The op travels over the WebSocket to a gateway, which routes it by doc_id to that document's session server.
The session server compares base_rev to its current revision. If the client is behind — it made this edit against rev 84120 but the server is at 84123 — the server transforms the incoming op against the three ops committed since 84120, adjusting positions so the edit still means what the author intended.
The server assigns the next revision (84124), applies the transformed op to its in-memory document, and appends it to the op-log durably.
Only after the durable append does the server ack the originating client (client_seq → rev 84124) and broadcast the transformed op to the other editors through their gateways.
Each receiving client transforms the incoming op against its own pending queue, then applies it. The originating client, on ack, pops that op from its pending queue.

Primary read path (open a document)¶

GET /api/v1/docs/{doc_id} loads the latest snapshot and the op-log tail after the snapshot's rev, materializes the text by replaying the tail (at most ~1,000 ops, ~2 ms), and returns { content, rev }.
The client opens the WebSocket at that rev (?from_rev=…). The session server streams any ops committed between the snapshot read and the subscription so the client catches up with no gap.
From then on the client is live: it receives broadcasts and sends edits over the same socket. Presence for the current collaborators is delivered on the side channel so the newcomer immediately sees everyone's cursors.

Storage choices¶

Op-log: append-only wide-column store (Bigtable / Cassandra / Spanner). Partition key doc_id, clustering key rev. Writes are pure appends and reads are ordered range scans within one partition — the ideal LSM workload. Replication across replicas/regions gives the durability an acknowledged edit demands. There are no updates and no cross-document joins, so nothing here wants a relational engine.
Snapshot: blob or document store, keyed by doc_id+rev. Snapshots are written rarely (every ~1,000 ops or on idle) and read once per open. A cheap object store is the right home; it is not on any hot path.
Presence: in-memory pub-sub (Redis) with short TTLs. Chosen for microsecond writes and automatic expiry. It is explicitly not durable — a cold presence store just means cursors briefly vanish and reappear on the next heartbeat, never data loss.

Scaling¶

Fan-out (the read-equivalent). The dominant cost is broadcasting each edit to every other editor. Within one document the session server does the fan-out; across documents, when a document's editors are spread over several gateways, the session server publishes each op once to a pub-sub topic per doc_id and the gateways holding subscribers deliver locally. To scale fan-out you add gateways; each edit is transformed once on the owner and delivered N times at the edges. Presence traffic scales the same way and is shed first under pressure because it is disposable.

Edit path. Session servers shard by doc_id across the fleet — a million active documents spread evenly because document ids hash uniformly. Adding session-server capacity means more documents per fleet, not more throughput for any single document. That last point is the hard limit: one document is one serialization point and cannot be sharded, because splitting its edits across two servers would produce two total orders and therefore divergence.

The 50-editor number, moved. One document with 50 concurrent editors is 250 ops/s inbound and ~12,250 msgs/s fan-out — comfortable for a single session server and a handful of gateways. Now push the scenario: if that same paragraph drew 5,000 simultaneous editors, inbound is 25,000 ops/s but fan-out explodes to 25,000 × 4,999 ≈ 125,000,000 msgs/s, which no single owner can serialize or ship. This is why real editors cap concurrent editors (Google Docs allows on the order of ~100 simultaneous editors) and split viewers onto read-only replicas fed by the same op stream. The resolution: keep true concurrent editing bounded at a size one owner can serialize, and turn the long tail of participants into read-only followers who receive broadcasts but do not write. Fifty editors sits deliberately inside that cap; five thousand forces the editor/viewer split.

Operational signals¶

The healthy signal is op round-trip latency — client send to server ack — sitting at p50 well under 50 ms and barely moving as editors join a document; a doc that gains 40 collaborators without moving round-trip latency is the system working as designed. The first metric to degrade under trouble is fan-out backlog: the session server's outbound broadcast queue or the gateway send-buffer depth climbing, which shows up as remote edits lagging before local editing feels anything. The misleading metric is average op latency across all documents — it stays flat because the overwhelming majority of documents have one or two idle editors, hiding a single hot document whose fan-out has quietly gone from 2 ms to 400 ms; watch per-document p99, not the fleet mean. The graph an experienced operator opens first during an incident is the per-document pending-transform queue depth on the owning session server paired with its broadcast-buffer depth: if edits are arriving faster than the owner can serialize-and-ship, that queue grows, and a growing single-document queue is the signature of a hot doc approaching its serialization ceiling.

Failure modes and resilience¶

Session-server crash. The in-memory document and transform state vanish, but the op-log is durable, so a replacement server acquires the document's lease, loads the latest snapshot, replays the op-log tail to rebuild the exact authoritative state and current rev, and resumes. Clients reconnect with their last-acked rev and resend unacknowledged pending ops, which the new owner transforms against anything it committed; per-client (client_id, client_seq) dedupe means a resend that was actually already committed is ignored rather than double-applied.
Split-brain (two owners for one document). The catastrophic case: two session servers each serialize edits, producing two divergent total orders that can never reconcile. Prevented by a single-writer lease on doc_id from a consensus store (etcd/Chubby-style). A server edits only while it holds a live, unexpired lease; on any doubt it stops accepting edits rather than risk a second order.
Client disconnect / offline editing. The client keeps editing locally against its pending queue while disconnected. On reconnect it replays its queued ops with their original base_rev; the server transforms them forward. Bounded offline windows reconcile cleanly; this is the one place a CRDT would be easier, and it is why local-first products choose CRDTs — but with a server present, OT handles the common brief disconnect fine.
The 50 editors, mid-storm. With 250 ops/s hitting one paragraph, the risk is not correctness — the total order guarantees convergence — but fan-out backpressure if a slow client can't drain 12,250 msgs/s. The lever is per-connection backpressure: the gateway coalesces or drops presence updates to that slow client first (cursors can skip frames harmlessly), and only if edits still can't drain does it disconnect and force a resync from a fresh snapshot, so one slow laptop can never stall the other 49.
Op-log write failure. The server withholds the ack — an edit is not acknowledged until it is durably logged — so the client keeps it in its pending queue and retries. The invariant "acked implies durable" is never weakened for latency.
Presence store outage. Cursors and collaborator lists go stale or blank; editing is completely unaffected because presence is off the durable path by design. It self-heals on the next heartbeat once the store returns.

Where this shows up in production¶

Google Docs — server-authoritative OT descended from the Jupiter model: a central server serializes every edit and transforms against concurrent ops, the exact spine of this design.
Google Wave — the ambitious OT system Docs' collaboration engine grew out of, and the source of much of the public OT literature.
Etherpad — the classic open-source OT editor; its "easysync" changeset format is a readable reference implementation of transform-and-broadcast.
Microsoft Fluid Framework / Office coauthoring — a total-order broadcast service orders ops centrally and fans them out, showing the "serialize then broadcast" pattern as an explicit ordering tier.
Figma — server-authoritative multiplayer where objects form a tree and each property is a last-writer-wins register, the CRDT-flavored contrast to text OT: convergence by commutative merge rather than transform.
Yjs / Automerge — CRDT libraries powering local-first and P2P apps, the alternative bet this design consciously does not take because it has a reliable central server.
Notion — block-structured documents synced as operations on blocks, applying the same op-stream idea above plain text.
Redis pub-sub at the edge — the ephemeral, TTL'd presence-and-fanout layer used exactly as here, kept strictly off the durable edit path.