What is NVMe?
NVMe, which stands for Non-Volatile Memory Express, is a high-performance interface for accessing solid state drives (SSDs) connected via the PCIe bus. Unlike traditional SATA SSDs that use the older AHCI protocol and SATA interface, NVMe was designed from the ground up for non-volatile memory like NAND flash, allowing for much higher speeds and lower latency.
Some key defining features of NVMe SSDs include:
- Utilizes PCIe interface for data transfer instead of SATA
- Offers up to 8x the bandwidth of SATA SSDs with PCIe 3.0, and up to 16x with PCIe 4.0 (Source 1)
- Much lower I/O latency and CPU overhead compared to AHCI
- Parallel command processing using multiple queues
- Designed for multi-core systems and solid state storage
The combination of PCIe bandwidth along with an efficient protocol designed for SSDs allows NVMe drives to reach incredible speeds not possible with SATA. Sequential reads can exceed 3,500 MB/s on high-end NVMe SSDs, while 4K random read performance can surpass 750K IOPS. This makes NVMe well suited for tasks like video editing, data analytics, gaming, and other bandwidth intensive workloads.
What is RAID?
RAID stands for “Redundant Array of Independent Disks.” It is a data storage technology that combines multiple disk drive components into a logical unit. The main goals of RAID are to provide increased data reliability and improved performance.
There are different RAID levels that provide various combinations of performance, redundancy, and efficiency:
- RAID 0 stripes data across multiple drives for faster performance but offers no redundancy.
- RAID 1 mirrors data across drives for redundancy but halves storage capacity.
- RAID 5 distributes parity data across drives, providing redundancy with more storage efficiency than RAID 1.
- RAID 6 adds an extra parity drive to RAID 5 for additional redundancy.
- RAID 10 combines striping and mirroring for increased performance and redundancy.
The key benefits of RAID include:
- Increased read and write performance by spreading I/O across drives.
- Redundancy to protect against drive failure and data loss.
- The ability to recover data if a drive fails.
NVMe RAID Compatibility
NVMe drives can be used in RAID configurations, but compatibility depends on your hardware setup. NVMe drives connect via PCIe slots rather than SATA ports like traditional SSDs. This means NVMe RAID solutions require a PCIe RAID controller card rather than a SATA RAID controller. Many server-grade motherboards include PCIe RAID support, allowing you to create bootable NVMe RAID arrays without a dedicated hardware RAID card.
According to HPE, NVMe drives are not natively supported by most hardware RAID controllers designed for SATA drives [1]. Instead, a PCIe add-in RAID card with NVMe support is required. However, newer RAID controllers may add NVMe RAID support via firmware updates. Always check compatibility lists before purchasing an NVMe RAID controller.
On the software side, most major RAID implementations like Microsoft’s Storage Spaces, Linux MDADM, and ZFS allow creating RAID arrays with NVMe drives. But booting from an NVMe RAID array requires UEFI motherboard firmware and OS support.
Performance benefits of NVMe RAID
NVMe RAID offers significant performance benefits over traditional SATA RAID in terms of both throughput and latency. NVMe drives utilize the high bandwidth of PCIe Gen 3 or Gen 4 lanes, with throughput up to 3500MB/s for sequential reads and 3000MB/s for sequential writes per drive. In comparison, SATA drives max out around 550MB/s. By combining multiple NVMe drives in RAID, even greater throughput can be achieved.
According to tests by DapuStor, their 4-drive NVMe RAID array using SSD7580 RAID controller reached up to 28.5GB/s sequential read and 21GB/s sequential write speeds[1]. For random access, NVMe also has much lower latency of under 100 microseconds, versus SATA latency of around 500-1000 microseconds. This results in significantly faster response times for random I/O.
Overall, the substantially higher bandwidth and lower latency of NVMe drives allows NVMe RAID to offer exponential performance gains over SATA RAID. Workloads involving large sequential transfers or heavy random I/O will see huge benefits with NVMe RAID.
Supported RAID levels for NVMe
NVMe drives are capable of supporting RAID 0, 1, and 10 configurations. According to Highpoint-Tech, their NVMe solutions powered by their RAID stack can support RAID 0, 1, and 10. Intel’s VROC (VMD NVMe RAID) also offers support for RAID 0 and 1 on their platforms1.
RAID 0 provides striping across drives for maximum performance but no redundancy. RAID 1 provides mirroring for redundancy but limits capacity to the size of a single drive. RAID 10 is a nested RAID combining mirroring and striping, providing both performance and redundancy but reducing overall capacity.
RAID 5 and 6, which use distributed parity, are not supported on NVMe due to performance limitations and rebuild times with parity calculations on the very fast NVMe drives. However, some vendors have developed proprietary implementations that allow RAID 5 and 6 for NVMe.
In summary, NVMe drives currently support RAID 0, 1, and 10 natively, providing both performance and redundancy options. More advanced RAID levels may be supported through proprietary controller implementations.
NVMe RAID Implementation
Setting up NVMe RAID involves both hardware and software components. At a high level, the basic steps are:
- Get compatible NVMe SSDs – Make sure your SSDs are compatible for NVMe and RAID. Opt for identical drives in terms of performance and capacity.
- Get a supported RAID controller – This can be a PCIe add-in card with RAID capabilities or integrated RAID support on the motherboard.
- Install NVMe SSDs – Mount the NVMe SSDs in PCIe slots on the motherboard or into the RAID card’s drive bays.
- Enable RAID mode in BIOS – Access the system BIOS and enable RAID mode for your NVMe controller.
- Create the RAID array – Use your RAID controller’s configuration utility to create the desired RAID array.
- Initialize and format the RAID volume – Once created, the RAID volume will need to be initialized and formatted just like a regular drive.
The main hardware requirements are an NVMe controller that supports RAID, at least two NVMe SSDs, and a way to connect them such as PCIe slots on a desktop motherboard or drive bays on a RAID card. Many server and workstation motherboards have integrated NVMe RAID support.
For software, you need RAID drivers installed and a utility to manage your RAID array. This usually comes with your RAID controller but may need to be downloaded from the manufacturer’s website. The utility is used to create, manage, and monitor the status of the RAID array.
Overall, NVMe RAID configuration involves getting compatible hardware, configuring RAID in the system BIOS, and using a RAID utility to initialize, create, and manage the RAID array. With the right components, creating NVMe RAID is very similar to setting up standard SATA RAID.
NVMe RAID Use Cases
NVMe RAID can provide significant performance benefits for certain use cases that demand high throughput and low latency storage. Some examples of optimal use cases for NVMe RAID include:
High-performance computing (HPC) – NVMe RAID can deliver the parallelism and speed needed for data-intensive HPC workloads like artificial intelligence, machine learning, and scientific modeling. RAID 0 and 10 are commonly used for the performance gains.
Real-time analytics – The low latency and high IOPS of NVMe RAID arrays can enable real-time analysis of huge datasets with minimal lag. Financial trading, fraud detection, and other time-sensitive analytics can benefit.
Media production – The bandwidth potential of NVMe RAID can facilitate smooth workflows for 4K/8K video editing, 3D rendering, and other media projects involving enormous files.
Virtualization – NVMe RAID helps reduce storage bottlenecks when running multiple virtual machines and instances. The improved performance can allow more VMs per host.
Database servers – OLTP databases need consistent low latency I/O for optimal transaction processing. NVMe RAID delivers faster access for more database operations per second.
Overall, applications and workflows that demand maximum throughput, IOPS, and the lowest possible latency stand to benefit greatly from implementing NVMe RAID storage.
NVMe RAID limitations
While NVMe RAID can provide significant performance benefits, it also comes with some notable drawbacks that users should be aware of before implementation:
Cost – NVMe SSDs are more expensive per GB compared to traditional HDDs or SATA SSDs. Building a RAID array with multiple NVMe drives can get very costly. The financial investment required makes NVMe RAID prohibitive for many home and small business users.
Heat – NVMe SSDs tend to run hotter than other drive types. Grouping multiple NVMe SSDs together in a RAID array can compound this heat issue. Proper cooling and airflow is critical when running NVMe RAID to prevent thermal throttling.
Compatibility – NVMe RAID requires CPU, chipset, firmware and OS support. Compatibility issues can arise with certain combinations of hardware and software. Not all NVMe SSDs play well together in RAID. Careful matching of components is advised.
Complexity – NVMe RAID is more complex to setup and manage versus standard single drive or SATA RAID configurations. IT expertise is recommended for optimal configuration and maintenance.
No redundancy – Many common NVMe RAID levels like 0 and 1 offer great speed but no data redundancy. Critical data may be lost if a drive fails.
Boot support – Booting from NVMe RAID arrays can be problematic depending on UEFI/BIOS and OS support. Getting NVMe RAID to successfully boot as the OS drive is not always straightforward.
Based on the high cost, limited compatibility, heat concerns, complexity and lack of redundancy, NVMe RAID will not make sense for many consumer and business use cases. The benefits of extreme speed come with substantial tradeoffs that users must carefully weigh before deployment.
Alternatives to NVMe RAID
While NVMe RAID offers impressive performance, it may not be the best solution for every use case. Here are some alternatives worth considering:
SATA SSD RAID – Setting up a RAID array with SATA SSDs is a more affordable option that can still provide performance gains over a single SATA SSD. SATA RAID is supported across a wider range of hardware and software solutions. The downside is slower speeds compared to NVMe RAID.
NVMe JBOD – NVMe Just a Bunch of Disks (JBOD) involves connecting multiple NVMe SSDs to a host without any RAID. This provides the full performance of each SSD individually. However, there is no redundancy. JBOD makes efficient use of NVMe bandwidth while keeping costs down.
Caching solutions – Using an SSD cache drive can boost performance of a larger HDD RAID array. The SSD cache stores frequently accessed data for faster reads and writes. Cash caching combines affordability of HDDs with improved responsiveness. However, the cache SSD represents a single point of failure.
The future of NVMe RAID
NVMe technology is still in the early stages of adoption, and as it becomes more widespread, we can expect to see continued innovation in NVMe RAID implementations. Some emerging technologies and expected trends include:
Increased adoption of NVMe over Fabrics (NVMe-oF) which allows NVMe storage to be shared over network connections. This enables the deployment of centralized, shared NVMe RAID arrays instead of localized RAID using directly attached NVMe drives. NVMe-oF makes NVMe RAID more feasible for enterprise environments. Kingston predicts NVMe-oF will be key for unlocking the full potential of NVMe RAID.
New RAID levels and implementations optimized specifically for NVMe’s ultra-fast performance. Rather than using traditional RAID levels like RAID 0, 1, 5 or 6, we may see new proprietary RAID implementations designed to maximize NVMe SSD speeds, IOPS, and endurance. Software-defined and hardware-accelerated RAID solutions will likely emerge to handle NVMe RAID efficiently.
In the data center, NVMe RAID will likely be combined with composable infrastructure and software-defined storage architectures for maximum flexibility. NVMe RAID pools can be composed, divided, and managed entirely through software.
On the client side, bootable NVMe RAID support will improve, allowing for ultra-fast OS and application drives. Motherboard, laptop, and pre-built vendors are expected to provide native NVMe RAID support more frequently.
As NVMe storage becomes ubiquitous, NVMe RAID will likely replace SATA RAID as the new storage performance standard. The incredible speed of NVMe SSDs demands a modern approach to RAID.