Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 65 ยท Case Studies: Classic Designs

Design a File Storage Service (Dropbox-style)

Design a cloud file storage and sync service: chunking, content hashes, deduplication, delta sync, metadata service, conflict handling and sharing, with Python.

A file storage and sync service such as Dropbox or Google Drive keeps a folder identical across your laptop, phone and the cloud. You edit a 2 GB video on one machine and the other devices catch up without re-uploading the whole file. Behind that magic are three ideas: split files into chunks, identify chunks by hash, and keep file metadata separate from file bytes.

Requirements

  • Upload, download, update and delete files up to tens of GB.
  • Automatic sync across devices; offline edits sync later.
  • Sharing with other users, with permissions.
  • File history and the ability to restore a previous version.
  • Reliable (no data loss, 11 nines durability) and efficient on bandwidth.
  • Scale: 500 million users, 100 million daily actives, 200 files each on average, average file 500 KB.

Estimates: 500M x 200 x 500 KB is about 50 PB of raw data before replication and deduplication. Metadata is tiny in comparison but much more frequently read and written, so it lives in a different system.

Architecture

flowchart LR
    C["Client app (watcher, chunker)"] --> API["API / sync service"]
    API --> META[("Metadata DB")]
    API --> NQ["Notification service"]
    C <--> BS["Block store service"]
    BS --> OBJ[("Object storage")]
    NQ --> C2["Other devices"]

Metadata and blocks travel on separate paths; clients upload chunks straight to the block store.

  • Block (chunk) store: holds immutable chunks keyed by hash, in object storage, replicated across zones.
  • Metadata DB: stores users, folders, files, the ordered list of chunk hashes per file version, and permissions. It needs strong consistency per user (relational, sharded by user ID).
  • Sync/notification service: tells other devices when something changed, by long polling or WebSockets.

Chunking and content hashes

Instead of treating a file as one blob, the client splits it into chunks of about 4 MB and computes a hash (SHA-256) of each. A file version is then just an ordered list of hashes. This delivers three wins:

  • Delta sync: edit one part of a big file and only the changed chunks upload.
  • Deduplication: two files (or two users) with the same chunk store it once.
  • Resumable uploads: after a dropped connection, upload only the chunks the server does not have.
Python Chunking, hashes and deduplication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One edited chunk caused one chunk of upload, and the duplicate file cost nothing. One caveat: fixed-size chunking breaks down when you insert bytes near the start, because every later chunk shifts. Production systems use content-defined chunking (a rolling hash picks boundaries from the data itself) so an insertion changes only nearby chunks.

Metadata and versions

Metadata rows look like file(id, owner, path, version) and version(file_id, n, [hash list], modified_at). Because chunks are immutable, keeping history is cheap: an old version is just an older hash list. Restoring means pointing the file back at it. Deleting a file marks it deleted; a garbage collector later removes chunks that no version references.

Python A metadata store with versions and conflict detection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rule is optimistic concurrency: a commit states which version it is based on. If the server's version moved on, it does not silently overwrite; it keeps a conflicted copy and lets the user decide.

Sync flow

  1. A file watcher on the client notices a change.
  2. The client chunks and hashes the file, asks the block store which hashes are missing, uploads those (in parallel, compressed, optionally encrypted).
  3. The client commits the new hash list to the metadata service with its base version.
  4. The metadata service records it and publishes an event.
  5. Other devices receive the event, fetch the new hash list, download only the chunks they lack, and rebuild the file.

More design points

  • Durability: store each chunk in at least three places (or use erasure coding to cut the storage overhead to about 1.5x), and verify hashes periodically to catch bit rot.
  • Hot and cold tiers: move rarely accessed chunks to cheaper storage.
  • Security: encrypt in transit and at rest; for shared folders enforce permissions in the metadata service, and issue short-lived signed URLs for block access.
  • LAN sync: devices on the same network can exchange chunks directly.
  • Bandwidth: throttle, compress, and skip temporary files.
  • Large teams: shard metadata by user or team, never by path alone.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why identify chunks by a content hash?

    Equal content gives an equal hash, enabling dedup and delta sync.
  2. 2. What problem does content-defined chunking solve?

    Boundaries derived from the data stay stable when bytes are inserted.
  3. 3. Two devices edit version 3 of a file while offline. What should the server do on the second commit?

    Optimistic concurrency prevents silent data loss.
  4. 4. Why are metadata and blocks stored in different systems?

    Metadata is small, hot and needs strong consistency; blocks are huge, immutable and fit object storage.

Summary

  • Split files into chunks, name chunks by hash, and store them immutably in object storage.
  • A file version is an ordered list of chunk hashes held in a consistent metadata database.
  • Hashing gives delta sync, deduplication and resumable uploads; content-defined chunking handles insertions.
  • Use base versions to detect conflicts, and keep history cheaply through immutable chunks.
  • Notify other devices by events and replicate chunks for durability.

Next: Design a Video Streaming Service