Networking

Storage Fabric

A storage fabric is the network infrastructure that connects compute hosts to disaggregated storage resources. Modern storage fabrics use NVMe-oF over Ethernet (NVMe/TCP or NVMe/RDMA), replacing proprietary Fibre Channel SANs with IP networks that are cheaper, more scalable, and cloud-compatible.

NVMe Storage

Storage Fabric Technologies

Several technologies can form a storage fabric, each with different latency, cost, and complexity profiles:

  • NVMe/TCP over Ethernet — The most accessible option. Runs on standard 10/25/100GbE switches and NICs. No special hardware. Latency 25–40µs.
  • NVMe/RDMA (RoCE) — RDMA over Converged Ethernet. Requires priority-flow-control (PFC) on switches and RDMA-capable NICs. Latency sub-20µs.
  • InfiniBand — Dedicated HPC fabric with ultra-low latency. Expensive and proprietary.
  • Fibre Channel (FC) — Legacy SAN fabric. Reliable but high cost, complex management, not cloud-compatible.

Fabric Design Considerations

For NVMe-oF deployments, network design matters more than for traditional block storage. Key considerations:

  • Bandwidth — NVMe-oF storage nodes can saturate 25–100GbE links. Use link aggregation or multi-path for initiators with high throughput requirements.
  • Latency — Use low-latency switches (cut-through forwarding). Avoid deep buffer switches for latency-sensitive NVMe-oF traffic.
  • Isolation — Dedicated storage VLANs or a separate storage switch fabric prevent data traffic from impacting storage I/O.