Skip to content

00. Design a File Sync Service

~20 min read · Level: intermediate–advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem

A file sync service keeps a folder identical across every device a person owns, and across everyone they share it with. This is the product behind Dropbox, Google Drive, and OneDrive. You drop a file into a watched folder on your laptop, and moments later it appears on your phone, your desktop, and your colleague's machine — and when you edit it anywhere, that edit propagates everywhere without you thinking about it. The magic the user perceives is "my files are just there, everywhere." The engineering reality is that several independent devices, each mutating a shared tree of files while frequently offline, must all converge to the same state without losing a byte anyone committed.

What makes this a genuine system-design problem is not moving bytes — HTTP does that. It is that every device holds its own copy and edits it independently, so the system is really a distributed-state-convergence engine wearing a folder's clothes. Two devices can edit the same file in the same second. A device can be offline for a week and come back with a stack of changes. A 1 GB file can get a one-line edit, and re-uploading the whole gigabyte to move a few kilobytes of actual change would be indefensible. The design lives or dies on how it represents changes, orders them, and reconciles them.

To keep the reasoning concrete, thread one scenario through the whole study: a user adds a 1 GB video to a synced folder, and then a one-line change is made to it on two devices at the same instant. The video chunks into 256 blocks of 4 MB, all of which upload the first time. The edit rewrites exactly one of those blocks, so only 4 MB should cross the wire instead of 1 GB — a 256:1 saving that only block-level thinking buys you. And because the edit happened on two devices at once, the system must pick a winner deterministically, preserve the loser's work rather than silently discarding it, and leave every device agreeing on the result. That single file, and that single collision, exercise every decision below.

Functional requirements

  • Sync: any change in a watched folder on one device (add, edit, delete, move, rename) propagates to all of that user's devices and all shared collaborators.
  • Efficient upload: only the changed portion of a file transfers, not the whole file. Editing one block of a 1 GB video moves 4 MB, not 1 GB.
  • Deduplication: content that already exists in the system — the same block, whether from the same user or a different one — is never stored or uploaded twice.
  • Versioning: keep prior versions of a file so a user can recover an overwritten or deleted file for some retention window.
  • Sharing: a folder can be shared with other users, who then sync it as if it were their own.
  • Conflict handling: when two devices edit the same file concurrently, resolve to a single deterministic outcome without data loss.

De-scoped for this round, and worth naming so the interviewer hears it as a choice: real-time collaborative content merging inside a document (the Google-Docs-style character-by-character merge — that is the sibling "collab editor" study, and for opaque files like a video it is not even possible), full-text search over file contents, granular per-file ACLs beyond folder-level sharing, and client-side end-to-end encryption. Each is real; none reshapes the core sync engine.

Non-functional requirements

The dominant constraint is convergence under concurrency with zero acknowledged-write loss. Everything else bends around it. Blobs are the easy half of this system — they are immutable, content-addressed, and trivially cacheable and replicable. The hard half is the metadata: the file tree and the ordered history of operations on it, which must be strongly consistent so that every device, applying the same ordered log of changes, lands on byte-identical state.

  • Consistency: the metadata — the mapping from paths to file versions to block lists, and the order of commits — must be strongly consistent and totally ordered per namespace (a namespace being one user's root or one shared folder). Blob storage can be eventually consistent; the file tree cannot.
  • Durability: once the service acknowledges a commit, that version must never be lost. A user who saw the spinner turn to a checkmark has been promised their bytes are safe.
  • Availability: reads and local edits must work offline and reconcile later; the sync path should tolerate a metadata replica or a block region failing without blocking the user's local work.
  • Efficiency: bandwidth and storage are first-class. Delta transfer and dedup are not optimizations bolted on later — they are the reason the economics work at all.
  • Sync latency: a change should become visible on other online devices within a few seconds, not instantly. This is a soft target, unlike the hard consistency requirement on ordering.

Scale estimation

Assume a mid-large service: 100 million registered users, 10 million daily active. Each active user makes on the order of 20 file mutations per day (edits, adds, moves), each recorded as a metadata commit.

Commits work out to 10M × 20 = 200M commits/day, or 200M / 86,400 s ≈ 2,300 commits/second on average. Apply a 10× peak factor for time-zone-aligned working hours and call it ~23,000 metadata commits/second at peak. That is the write load the metadata service must serialize and order, and it is the number that forces sharding by namespace rather than one global log.

Each active device holds one long-lived connection to a notification service so it learns about changes without polling. That is ~10 million concurrent connections; at ~50,000 connections per notification node, roughly 200 notification nodes carry the fan-out.

For storage, take an average of 2 GB stored per user: 100M × 2 GB = 200 PB of logical data. Block-level dedup across and within files reclaims on the order of 30% in the common case (shared installers, re-sent attachments, duplicated blocks inside files), so physical storage is closer to ~140 PB — and for a genuinely shared asset the win is far larger, which the scenario shows below. At a 4 MB block size, 200 PB is roughly 200 PB / 4 MB ≈ 50 billion blocks, each keyed by its content hash. That block count, not the byte count, is what the blob store's index must handle.

Upload bandwidth is where delta sync earns its keep. Say each active user pushes ~50 MB of genuinely changed blocks per day: 10M × 50 MB = 500 TB/day ≈ 5.8 GB/second average, and ~58 GB/second at peak. Now weigh that against the naive alternative. Our scenario's one-line edit to a 1 GB video, under whole-file upload, would move 1,024 MB; under block-level sync it moves 4 MB — a 256:1 reduction. Multiply that ratio across every large-file edit in the system and the difference is not a tuning knob; it is whether the upload tier is affordable at all.

API sketch

POST /commit                          # propose a new version of a file
  body: { namespace_id, path, parent_cursor,
          blocklist: [<sha256>, ...], size, mtime, device_id }
  200:  { committed_cursor, version_id }
  409:  { conflict: true, current_cursor, current_version }   # parent was stale

POST /blocks/probe                     # ask which blocks the store is missing
  body: { hashes: [<sha256>, ...] }
  200:  { missing: [<sha256>, ...] }    # upload only these

PUT  /blocks/{sha256}                  # upload one 4 MB block (content-addressed)
  200:  stored (idempotent; a re-PUT of existing content is a no-op)

GET  /delta?namespace_id&since_cursor  # pull all changes since a cursor
  200:  { changes: [{path, version_id, blocklist, ...}], new_cursor }

GET  /longpoll?namespace_id&cursor     # block until namespace advances past cursor
  200:  { changed: true, new_cursor }

Solutioning

Start from the requirement that a one-line edit to a 1 GB file must move kilobytes, and the whole shape follows. You cannot treat a file as an opaque blob you overwrite; you must split every file into fixed-size blocks (4 MB), hash each block, and represent the file as an ordered list of block hashes. Now a file version is just a recipe of hashes, the blocks themselves live in a content-addressed store keyed by hash, and "what changed" is a set difference between two hash lists. The scenario's edit changes exactly one block, so the new version's recipe differs from the old in one position, the client uploads the one new block, and the other 255 are already present. The reframing worth saying in the room: syncing a large edit is not a bandwidth problem; it is a diffing problem — solve the diff and the bandwidth takes care of itself.

Content-addressing hands you deduplication for free, which is the first defining tradeoff. Because a block's key is its content hash, two identical blocks — the same video uploaded by two employees, the same email attachment saved by a hundred people — collapse to one stored object automatically; the second uploader's probe call reports zero missing blocks and transfers nothing. The tension is that dedup buys storage and bandwidth at the cost of complexity in the delete path: you can no longer delete a block just because one file stopped referencing it, since a thousand other files might still point at it. That forces reference counting and careful garbage collection with a grace period, and a hash-collision policy. The resolution is to accept that complexity deliberately, because at 50 billion blocks the ~30% storage reclaim (and the 256:1 bandwidth reclaim on edits) is worth far more than a simpler GC would save — but to confine the complexity to an offline sweep, never the hot path.

The second defining tradeoff is strongly consistent metadata against eventually consistent blobs, and the trick is to split the system exactly along that seam. The file tree, the version history, and the commit order must be strongly consistent — two devices editing the same file must serialize into one order that every device replays identically — so metadata lives in a sharded transactional store, partitioned by namespace, where each namespace has a single serialization point and a monotonically increasing cursor. Blocks, being immutable and content-addressed, have no ordering to get wrong; a block either exists under its hash or it does not, so the blob store can be eventually consistent, geo-replicated, and erasure-coded for cheap durability. This is the split that makes the numbers work: the expensive-to-coordinate part (metadata) is small and shardable, while the enormous part (140 PB of blocks) needs no coordination at all.

That leaves the third tradeoff, conflict resolution strategy, which the scenario puts under a spotlight: two devices commit a new version of the same file at the same instant. Both computed their new block list against version N and both try to commit as version N+1. The metadata service uses optimistic concurrency — each commit names its parent_cursor, and the namespace advances only if that parent still matches the head. The first commit to reach the serialization point wins and moves the head to N+1; the second arrives with a now-stale parent and is rejected with a conflict. Rather than discard the loser's bytes, the service materializes them as a conflicted copy — a sibling file like video (Device-B's conflicted copy).mp4 — so both edits survive and every device converges on the same two files. The choice here is deliberate: for opaque files you cannot merge content, so the honest strategy is "one deterministic winner keeps the name, the loser is preserved beside it, and nothing is ever silently lost" — not a best-effort last-writer-wins that quietly drops an edit. The following files take these three decisions down to components (HLD) and then to schemas, chunking math, and the concurrency corners where sync actually breaks (LLD).