01. File Sync — High-Level Design¶

~15 min read · Part 2 of 4 (Overview → HLD → LLD → Q&A)

This file turns the solutioning narrative into concrete boxes and the flows between them. Read the architecture top to bottom, follow a write and a read through it, then watch what happens when the pieces fail — including the two-device edit from the scenario.

Architecture¶

   ┌───────────────────────────────────────────────────────┐
   │  Client sync engine  (laptop / phone / desktop)        │
   │  · watches folder   · chunks into 4 MB blocks          │
   │  · hashes blocks    · local file→block DB + cursor     │
   └───────┬───────────────────────────────┬───────────────┘
           │ metadata (commit / delta)      │ blocks (probe / PUT)
           ▼                                ▼
   ┌───────────────┐                ┌────────────────┐
   │ Load Balancer │                │ Load Balancer  │
   └───────┬───────┘                └───────┬────────┘
           ▼                                ▼
   ┌───────────────────┐            ┌────────────────┐
   │  Metadata Service │            │  Block Service │
   │  (commit / delta) │            │  (probe / PUT) │
   └───┬───────────┬───┘            └───────┬────────┘
       │           │                        │
       ▼           ▼                        ▼
 ┌───────────┐ ┌────────────┐       ┌────────────────┐
 │ Metadata  │ │Notification│       │  Blob Store    │
 │  DB       │ │  Service   │       │ (content-      │
 │ (sharded  │ │ (long-poll,│       │  addressed,    │
 │  by ns,   │ │  fan-out)  │       │  erasure-coded │
 │  cursor)  │ └─────┬──────┘       │  by SHA-256)   │
 └───────────┘       │              └────────────────┘
        ▲            │ "namespace N advanced"
        │            ▼
        └──── other devices pull /delta

Read it as two rails that meet only in the client. The block rail on the right moves immutable, content-addressed bytes: the client asks the Block Service which hashes are missing, uploads only those, and they land in a blob store keyed by content hash where dedup happens automatically. The metadata rail on the left moves the small, ordered, strongly consistent record of what the file tree looks like: a commit names a file's new block list and its parent cursor, the Metadata Service serializes it into the namespace's ordered history, and the Notification Service tells every other device that the namespace advanced so they pull the delta. The deliberate asymmetry is the whole design — the metadata rail is tiny and carefully ordered, the block rail is enormous and needs no ordering at all.

Components¶

Client sync engine. The most sophisticated component, and the one candidates under-weight. It watches the filesystem for changes, splits each changed file into 4 MB blocks, hashes them, and diffs the new block list against the last-synced version held in a local database. It uploads only missing blocks, commits the new block list against its last-known cursor, and in the other direction applies deltas pulled from the server onto the local disk. It also holds the offline queue: edits made with no network sit in the local DB until connectivity returns.

Metadata Service. The source of truth for the file tree. It handles commit (propose a new file version) and delta (pull changes since a cursor), enforces optimistic-concurrency ordering per namespace via the cursor, detects conflicts, and materializes conflicted copies. Stateless itself; all durable state is in the Metadata DB.

Metadata DB. A sharded transactional store, partitioned by namespace_id, holding the file tree, version history, block lists, and the per-namespace monotonic cursor. Sharding by namespace means every operation on one folder hits one shard, giving a single serialization point and cheap ACID ordering without cross-shard coordination.

Block Service. A thin, stateless gateway in front of the blob store. Its probe endpoint answers "which of these hashes do you already have?" so the client uploads only genuinely new blocks, and its PUT verifies each block's content against its claimed hash before writing. It never overwrites — content addressing means a PUT of existing content is a no-op.

Blob Store. The content-addressed store for immutable 4 MB blocks, keyed by SHA-256, erasure-coded and geo-replicated for durability at low cost. This is where the 140 PB lives. It carries no notion of files, paths, or users — only opaque blocks under content hashes, which is exactly what lets it be eventually consistent and dumb.

Notification Service. Holds the ~10 million long-lived client connections and fans out a lightweight "namespace N advanced to cursor C" signal whenever a commit lands. It carries no file data — just a nudge that tells a device to pull the authoritative delta. Because it is only a hint, losing a notification costs latency, never correctness.

Primary write path (add or edit a file)¶

The sync engine detects a changed file, splits it into 4 MB blocks, and hashes each. For the scenario's 1 GB video that is 256 blocks; for its later one-line edit it is the same 256 hashes with exactly one changed.
The client calls POST /blocks/probe with the new block list. On first upload the store is missing all 256; after the edit it is missing exactly the one rewritten block.
The client PUTs only the missing blocks to the Block Service, which verifies each block's hash and writes it to the blob store. First upload: 1,024 MB. The edit: 4 MB.
The client calls POST /commit with the file's path, the new ordered block list, and its parent_cursor (the cursor it last synced). The Metadata Service checks that the namespace head still equals parent_cursor; if so, it writes the new version, advances the cursor, and returns the new cursor.
The Notification Service fans out "namespace advanced" to every other connected device for that namespace.
The commit is durable before the client sees success, so the checkmark is an honest promise.

Primary read path (receive a change on another device)¶

A device's long-poll on the Notification Service returns "namespace advanced to cursor C," or its periodic poll notices the cursor moved.
The device calls GET /delta?since_cursor=<last> and receives the list of changed files with their new block lists and the new cursor.
For each changed file, the device diffs the incoming block list against what it holds locally and calls probe/GET to fetch only the blocks it is missing — again, one block for the scenario's edit, not the whole file.
It reconstructs the file from its block list, writes it to disk atomically (write-to-temp-then-rename so a crash never leaves a half-file), and advances its local cursor to C.
If applying the delta collides with an un-synced local edit, the device does not clobber the local work — it defers to the same conflict path the server uses, so the two edits reconcile rather than one overwriting the other.

Storage choices¶

Metadata: sharded transactional store (MySQL/Postgres-class), partitioned by namespace_id. The access pattern is small, ordered, transactional writes and point/range reads within one namespace — exactly relational's strength. Partitioning by namespace localizes every commit to one shard, so the cursor's compare-and-advance is a single-row transaction with no distributed consensus. Cross-namespace operations (a file moving between shared folders) are the rare, expensive case and are handled explicitly.
Blocks: content-addressed object store, erasure-coded, geo-replicated. Immutable and keyed by content hash, so it needs no transactions, no locking, and only eventual consistency; erasure coding gives ~1.5× storage overhead for high durability instead of 3× full replication. This is the profile of S3 and of Dropbox's Magic Pocket.
Notification state: in-memory, per-node connection tables. Ephemeral by design; a node restart drops connections that clients simply re-establish, and no durable state is lost because the authoritative cursor lives in the Metadata DB.
Client-local: an embedded database (SQLite-class). Stores the file→block mapping, hashes, and last-synced cursor so the client can diff locally and work offline without re-reading and re-hashing the whole folder on every change.

Scaling¶

Metadata (write) path. The ~23,000 commits/second peak is spread across namespace shards; sharding by namespace_id means adding shards adds commit throughput linearly, and no single namespace's commits ever cross shards. The hot case is not total volume but a single shared namespace with many concurrent editors, which concentrates commits and optimistic-concurrency retries on one shard — handled by serializing that namespace's commits through a short per-namespace queue rather than letting them thrash the retry loop.

Block (write/read) path. The blob store scales independently of metadata and absorbs the ~58 GB/second peak upload by horizontal object-store scaling. Dedup does double duty here: on the scenario's edit it cuts the transfer 256:1 (4 MB not 1 GB), and on a genuinely shared asset it collapses storage — a 1 GB onboarding video synced by 1,000 employees is stored once, a 1,000:1 reclaim on that file rather than storing 1,000 GB.

Notification path. 10 million connections at ~50,000 per node is ~200 nodes; scaling is adding nodes and resharding the connection-to-namespace routing. Because the payload is a tiny nudge, per-node bandwidth is negligible — the constraint is connection count and fan-out latency, not throughput.

Operational signals¶

The healthy signal is sync convergence lag — the time from a commit landing to the last online device reflecting it — which should sit at a few seconds and barely move under load. The first metric to degrade under trouble is the optimistic-concurrency conflict/retry rate on a namespace: when many editors pile onto one shared folder, commits start failing their parent-cursor check and retrying, and that rate climbs before latency does. The misleading metric is aggregate upload bandwidth — it can look perfectly healthy while one hot namespace's p99 commit latency has quietly blown out, because the average is dominated by millions of quiet namespaces; watch per-namespace commit latency, not the fleet average. The graph an experienced operator opens first during an incident is the per-shard commit-cursor advance rate alongside the conflict-retry rate: a shard whose cursor is advancing slowly while retries spike is a hot shared namespace serializing on itself, and that pattern localizes the incident to one folder rather than the whole tier.

Failure modes and resilience¶

Two devices edit the same file at once (the scenario). Both probe/upload their changed block and both commit version N+1 with parent_cursor = N. The Metadata Service serializes them: the first advances the head to N+1 and wins; the second's parent is now stale, so it is rejected with 409 and its version is materialized as a conflicted copy. No bytes are lost, and because the serialization point is single per namespace, every device pulling the delta sees the identical outcome — one winning file plus one conflicted copy. The failure is contained by design, not patched after the fact.
Metadata shard outage. Commits to namespaces on that shard stall, and local edits queue on the clients until it recovers; reads can be served from a replica. This is the critical store, so it runs with synchronous replicas and fast failover — a metadata loss is unacceptable, whereas a metadata pause is merely painful. Blocks are entirely unaffected because they live on the other rail.
Blob store partial outage. New uploads for blocks that would land on the affected partition fail and retry; files whose blocks are already stored remain fully readable. Because blocks are immutable and content-addressed, retries are safe and idempotent, and geo-replication lets reads fail over to another region.
Notification Service outage. Devices stop getting nudges and fall back to periodic polling of /delta, so convergence lag rises from seconds to the poll interval — degraded latency, never lost or corrupted state, because the cursor in the Metadata DB remains authoritative.
Client crash mid-upload. Blocks uploaded but never committed become orphans referenced by nothing; they are reclaimed by the reference-counting GC sweep, not on the hot path. Because commit is the atomic point and it names a fully-uploaded block list, a half-uploaded file simply never becomes a version.
Conflict storm on a hot shared folder. Many editors thrash the optimistic-concurrency retry loop, spiking the conflict-retry rate. Mitigation is to serialize that namespace's commits through a short server-side queue so writers wait briefly rather than retrying blindly, trading a little commit latency for a bounded, orderly outcome.

Where this shows up in production¶

Dropbox — splits files into 4 MB fixed blocks stored content-addressed in Magic Pocket, so an edit re-uploads only changed blocks and identical blocks across users are stored once; the canonical instance of this design.
Git — content-addresses every object by hash, so identical blobs are stored once and a commit is a recipe of hashes; the same "file is a list of content hashes" model as our block lists.
rsync — uses a rolling checksum to transfer only the changed byte ranges of a file, the delta-transfer ancestor of block-level sync and the reason a one-line edit doesn't re-send the file.
Google Drive — versions opaque files at the file level (like our conflicted copies) while Google Docs merges content via operational transforms — the fork in the road between "can we merge?" and "we can only preserve both."
Apple iCloud / CloudKit — hands clients a per-zone change token that they present to pull deltas, the same cursor-and-delta pull model as our namespace cursor.
restic / borgbackup — dedup backups with content-defined chunking, refcounted blocks, and grace-period GC, exercising exactly our delete-path complexity at rest.
Amazon S3 — the immutable, eventually-consistent, erasure-coded object-store profile our blob rail wants; content-addressed keys and no cross-object transactions.
Syncthing — a peer-to-peer file sync that exchanges blocks and detects conflicts with version vectors, showing the same block-plus-conflict model without a central metadata authority.