Persistent memory is one of the new features brought into the latest version of vSphere to further improve the flexibility of the platform. This term does not only apply to VMware products as it makes use of a new type of memory, the NVM. It is comparable, in a way, to an ultra-fast SSD storage. It is another layer of storage that sits between a hard disk or flash Nand and memory to improve performances.
The main different layers of storage of a server include CPU cache, DRAM, and disk (SSD or HDD), each being slower than the previous one. Apart from the huge performance gap (bandwidth and Latency), the big differentiator between DRAM and disk lies in the fact that the former is a volatile memory, meaning it won’t retain whatever data it was storing after a power cycle.
What is NVDIMM?
Non-volatile Memory (NVM or NVDIMM) is a revolutionary technology that will further extend the performances of our servers. It was developed as a type of RAM that retains its content as opposed to traditional DRAM. These types of modules can dump the content of the DRAM into the NAND flash component. NVDIMM operates in regular memory slots to be as close as possible to the processor and handles critical data at speed and latency of DRAM!
“NVDIMM is a nonvolatile persistent memory solution that combines NAND flash, DRAM, and an optional power source into a single memory subsystem. The solution delivers DRAM-like latencies and can back up the data it handles, providing the ability to restore quickly if power is interrupted.”
NVDIMMs come in the form of a memory module made of traditional DRAM with Flash NAND components located on the dimm itself acting as a tier of storage. So instead of sending straight to disk and rely on the access times of the disk and the SAS cable, the content of the RAM can be stored in its persistent module for much faster access by the CPU when needed. NVDIMMs are supposed to have more than 10 times the density of DRAM and 100 times the performances of SSDs. The latency being DRAM like is measured in nanoseconds! Most of these modules also embed a capacitor to allow the RAM to offload to persistent memory in case of electrical failure.
Having that extra layer of storage will greatly improve performance by reducing latency times to access the data and relieve the backend storage from IOs that will be handled by the modules instead.
| Storage Technology | Data Access Time |
|---|---|
| 15K SAS Disk | ~ 6,000,000 ns |
| SATA SSD | ~ 120,000 ns |
| NVMe SSD | ~ 60,000 ns |
| DDR4 NVDIMM-N | ~ 150 ns |
There are multiple benefits of persistent memory:
- Better performances with direct memory access and improved CPU and memory utilization
- Greater capacity for in-memory workloads like database caching and analytics data sets
- Relieve the backend storage (SSD/HDD) from a great deal of IOPS, increasing, in turn, their lifespan
Virtual machines like databases that require high bandwidth and low latency will be the first workloads to benefit from this new type of storage.
vSphere 6.7 implementation
In vSphere 6.7, persistent memory can be leveraged by hosts and virtual machines whether an application has been written to support it or not. Once a memory module is added to a host, it will be picked up by vSphere and appear as “Persistent Memory” next to “Memory” (DRAM).
The NVDIMM capacity of the modules is presented as a local datastore on the host. This involves that any VM (even legacy OS) can take advantage of it and no compatibility or guest OS changes are required. vSphere currently supports 2 modes of accessing persistent memory:
vPMEMDisk (Right on the diagram)
With vPMEMDisk, vSphere allocates a piece of the PMEM datastore and presents it to the virtual machine as a virtual disk that can be mounted on an SCSI controller like any other disk. No change is required in the guest, it doesn’t need to be compatible with NVDIMM and the VM doesn’t have to be compatible with 6.7.
In order to assign a disk to a PMEM datastore, a “hostlocal PMem default storage policy” will be applied to it. Note that once the policy is applied you cannot change it, you will have to migrate or clone it to do so.
vPMEM (Left on the diagram)
This mode is a bit less easy going as it will present the NVDIMM capacity to a VM as a “virtual NVDIMM device”. In this configuration, the guest Operating System must be compatible with NVM technology (Windows Server 2016 and Linux Kernel 4.2+) and the VM must be compatible with 6.7 (version 14). Instead of storing the virtual disk of the VM on the local PMEM datastore, vSphere will store the virtual NVDIMM device.
In this model, applications can experience the highest performance gain by having a direct and uninterrupted access to the NVDIMM hardware, as opposed to the previous mode where the VM only benefits from a really fast virtual disk.
Considerations about persistent memory
In either case, note that the PMem datastore is only used to store virtual NVDIMM devices and traditional virtual disks of a VM. Not that you would want to but you can’t store other files like vmx, log, iso… The only actions available on the datastore are monitoring statistics that will be available. Only one local PMEM datastore can exist per vSphere host.
Also, be aware that you must evacuate all the VMs from the host before placing it in maintenance mode.
After a VM is powered on, the PMEM resource is reserved on the host regardless of its future power state. The reservation is lifted when the VM is migrated or removed.
Migrating a PMEM enabled VM to another host involves using storage vMotion because it uses a local datastore, of course, the destination host must have enough PMEM resources to satisfy the VM’s provisioned capacity. However, a virtual machine with a vPMEMDisk can be migrated to a host with no PMEM resource, as opposed to those using vPMEM.
Supported devices
As of this time, only Dell and HP offer support and available devices to use persistent memory.
As opposed to Dell, HP proposes what they call HPE Scalable Persistent Memory in their DL380 Gen10. These modules aren’t referred to as a type of memory but as an “integrated storage solution that runs at memory speeds with terabyte capacity” and can scale to up to 1TB.
Conclusion
This Persistent memory technology is still very new in the VMware ecosystem and we will learn a lot more about it at the VMworld 2018. However, we can already speculate on where this is going. As the capacity of NVDIMM devices gets to reach 1TB (HPE), the DRAM components become more of a caching tier to the NVDIMM module like Flash NAND was to HDD a few years ago.
Chances are you don’t have a host with supported NVDIMM but you can already test the feature by manually reserving a percentage of a host’s memory to fake persistent memory, more info here.
As time goes by and prices come down, this new type of storage will surely become a game changer in how we deal with server IOPS and latency.
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
