NVMe (Non-Volatile Memory Express)
NVMe is a storage access protocol and interface specification designed from the ground up for solid-state storage. It replaces legacy SCSI-derived protocols with a streamlined command set that exploits the parallelism of NAND flash and 3D XPoint memory.
How NVMe Works
NVMe communicates over PCIe (Peripheral Component Interconnect Express), the same high-speed bus used for graphics cards. A host CPU writes NVMe commands directly into memory-mapped I/O registers on the SSD controller — no SCSI translation layer, no legacy driver stack.
Key advantages over SCSI/SATA
- 64,000 queues × 64,000 commands — vs AHCI's 1 queue × 32 commands. Modern SSDs serve thousands of concurrent operations without head-of-line blocking.
- Simplified command set — NVMe uses 13 base commands vs hundreds in SCSI. Less CPU overhead per I/O operation.
- Sub-100µs latency — on a local PCIe SSD, NVMe achieves 10–50µs end-to-end latency vs 100–500µs for SATA or SAS.
- Namespace model — a single NVMe controller can expose multiple independent namespaces, enabling multi-tenant storage on one device.
NVMe over Fabrics (NVMe-oF)
NVMe was originally a local bus protocol. NVMe-oF extends it across a network fabric (TCP, RDMA, or Fibre Channel), preserving the full command set and parallel queue model over the wire. This enables disaggregated storage — compute and storage scale independently while preserving near-local latency.
NVMe in Linux
The Linux kernel has supported NVMe since 3.3 (2012). The nvme kernel module handles local PCIe NVMe devices; nvme-tcp, nvme-rdma, and nvme-fc handle fabric transports. The nvme-cli userspace tool manages all NVMe devices and NVMe-oF connections.