S3 is not a filesystem
S3 is an excellent object store. Using it as if it were a filesystem works until it doesn’t — the moment you have millions of files, the hard problems move away from storing bytes.
The source
siddontang (TiDB), captured 2026-10-06 — a founder’s “very reasonable question” (why not just use S3?) and the answer: because S3 is not a filesystem. And the kicker: “You can build all of these around S3. But then… you’re building a filesystem 🙂” — which is exactly how TiDB thinks about TiDB Filesystem.
What an object store gives you
Flat namespace of immutable objects; PUT/GET/DELETE/LIST by key; effectively unlimited scale, 11-nines durability, cheap bytes at rest. For blobs — backups, media, datasets, artifacts, log segments — this is the right tool (see Open source backup software for the lane where S3 is the answer).
What it doesn’t (the four problems that emerge at millions of files)
- Metadata and namespace — no directories, no inodes, no per-file mtime/owner metadata beyond a handful of object tags; LIST over millions of keys is slow and rate-limited (and prefix-listing is not a tree walk).
- Lots of small-file reads/writes — per-object request costs dominate; latency per op vs. batch; no appends (full rewrite to change a byte).
- Rename, permissions, locking, concurrent mutations — rename = copy+delete (atomic nowhere); no POSIX locks; conditional writes exist but multi-object transactions don’t; consistency is per-object, not cross-key.
- Filename / metadata / content search — no indexes; you build them, which means… metadata storage, which is problem 1 again.
The pattern: every workaround compounds into reimplementing filesystem machinery on top of S3 — metadata service, journal, locking layer, index. At that point S3 is the disk, and you’re the FS.
Where this shows up
- LakeFS/JuiceFS/Prefix-less stores: mount-S3-as-FS products all carry the same asterisks — metadata caching, rename semantics, small-file pain.
- TiDB Filesystem (the tweet’s context): build the FS properly, use object storage as the block layer.
- git-annex/restic/borg style backup (Open source backup software): they chunk and index locally, treating S3 as dumb durable bytes — the correct division of labor.
- HuggingFace/Hub-scale artifact stores: content-addressed blobs + separate metadata DB — object store discipline, not FS emulation.
- The rmoff capture (S3-compatible MinIO alternatives) is the inverse question: choosing the object store, not mistaking it for a FS.
Rule of thumb
- Bytes that are written once, read as a whole, keyed by name → object store, directly.
- Anything users open, rename, lock, partially write, or search by metadata → filesystem semantics required; either a real FS or a purpose-built layer over S3 — and know that you’re now maintaining that layer.
- The failure mode is silent: a few thousand files works fine on naive S3 layouts; the wall arrives at scale, in production, as latency.
Vault kin
- Change Data Capture — cross-key consistency needs are the same class S3 refuses to sell you.
- Performance engineering — small-object request economics; LIST is O(keys).
- Local archives & crawlers — SQLite-as-index over dumb storage is the same pattern at personal scale.