NVMe vs Object Storage: Block vs S3
NVMe block storage and S3-compatible object storage are not competing technologies — they solve fundamentally different problems. NVMe provides a raw device interface for the OS to format and mount; object storage provides an HTTP key-value API for arbitrary-size blobs. Using the wrong type for your workload leads to either severe performance problems or runaway cost.
Quick Verdict
- Use NVMe block storage for databases (PostgreSQL, MySQL, Redis), VMs, Kubernetes PVCs, OS disks, any workload requiring random I/O < 100ms.
- Use object storage for media files, ML training datasets, backup/archival, data lake ingestion, static website assets.
- Use both in tiered architectures: hot data on NVMe, cold/archival data in object storage.
NVMe vs Object Storage: Key Differences
| Attribute | NVMe Block Storage | Object Storage (S3) |
|---|---|---|
| API / Interface | Block device (/dev/nvme0n1) | HTTP REST (GET/PUT/DELETE) |
| Latency (read) | 10–40µs | 1–100ms (first byte) |
| Random 4K IOPS | 500K–7M | ~100–1,000 (request-based) |
| Consistency model | Strong (block-level atomic) | Eventual (S3) or strong read-after-write (AWS S3 2021+) |
| Partial update (4K) | Yes (block I/O) | No — must re-upload entire object |
| Mount as filesystem | Yes (ext4, xfs, etc.) | No (FUSE mounts exist but are slow) |
| Max object/volume size | Up to 64TB per volume | 5TB per object (AWS S3); unlimited total |
| Cost (hot tier) | $0.10–$0.20/GB/month (cloud) | $0.02–$0.025/GB/month (S3 Standard) |
| Scalability | Manual volume provisioning; NVMe-oF pools scale horizontally | Virtually unlimited; auto-scales |
| Concurrent writers | One mount point (shared via POSIX or NVMe-oF) | Unlimited concurrent clients |
Why Databases Cannot Use Object Storage
Relational databases (PostgreSQL, MySQL, SQLite) require three things that object storage cannot provide:
- Partial updates — a database modifies 8KB pages inside a file. Object storage has no way to update 8KB inside a 1GB object without re-uploading it.
- Strong consistency — after a write commits, any read must see the new value immediately. Eventual consistency models cause data corruption under concurrent load.
- Sub-millisecond latency — even a 1ms read latency means a query touching 1,000 rows takes at least 1 second in I/O alone.
Some databases (CockroachDB, TiDB, Neon) use object storage as a write-ahead log or remote REDO log — not as the primary data path. The hot path still goes through local or NVMe-oF block storage.
Object Storage Strengths
Object storage wins decisively for use cases where data is written once and read sequentially as a whole object:
- ML training datasets — 100GB Parquet files streamed to GPUs. MinIO or S3 at 10–30 GB/s aggregate throughput beats even NVMe for this access pattern.
- Media and video — video files are read start-to-finish; no random access needed.
- Backups and snapshots — immutable objects with versioning; the PUT-once model is a feature, not a bug.
- Data lakes — petabyte-scale cold analytics where query latency of seconds is acceptable.
- Static web assets — images, JS, CSS with CDN in front; no filesystem needed.
Tiered Architecture: NVMe + Object Storage Together
Modern cloud-native architectures use both storage types in a tiered model:
The storage tiering decision is driven by access frequency and latency requirements, not by storage capacity alone. See the disaggregated storage glossary entry for more on shared NVMe-oF pool architectures.