When moving from the physical world over to virtual infrastructure, there are several sizing metrics that administrators may need to revisit when it comes to the amount of virtual hardware assigned to production virtual machines. Running production workloads on top of virtual infrastructure can provide several advantages over running physical servers in terms of efficiency and high availability. However, there are also several key areas that need to be considered in terms of virtual machines running on top of physical hypervisor hosts. Microsoft Windows Server Hyper-V provides a great platform for virtualized workloads. Hyper-V administrators need to pay attention to the design of Hyper-V clusters, virtual machine workloads, and the placement and configuration of the virtual hardware sizes per virtual machine. One of the key areas when looking at Hyper-V Cluster design is the virtual machine memory configuration.
In this post, we will take a closer look at virtual machine memory and key considerations needed to ensure the desired virtual machine performance of a particular virtual machine or set of virtual machines.
Virtual Machine Memory Considerations
There are many considerations that Hyper-V administrators need to make when thinking about memory allocations for Hyper-V virtual machines. These include the following considerations:
- Physical vs. Virtual memory allocations
- NUMA node considerations with Hyper-V virtual machines
- Dynamic Memory and Minimum RAM
Let’s take a closer look at each one of these areas of memory considerations when configuring a Hyper-V virtual machine.
Physical vs. Virtual memory allocations
Even though RAM prices have gone up considerably in the past couple of years, in today’s world of hardware, memory prices are still relatively cheap when considering the overall expenditure of a physical host. In the past when organizations built out physical hosts that would only be serving out physical server instances, organizations often configured the server with more RAM than it really needed without any real dire consequences to overall cost and/or performance.
However, virtualization changed many of the paradigms that existed when configuring physical servers with certain hardware configurations – one of those being memory. When thinking about configuring a number of virtual machines on a physical Hyper-V host, it is no longer a good idea to simply throw extra memory on a virtual machine as this can drastically impact both cost and performance with the so-called “wide VM” configuration. In the virtual world, assigning additional memory to a virtual machine really has no beneficial effect aside from wasting useable resources that could be used to provision other virtual machines.
Microsoft has a very helpful tool in this regard called the Microsoft Assessment and Planning Toolkit that makes it easy to assess your current IT infrastructure for a variety of technology migration projects including virtualizing resources.
Virtualization has made administrators much more aware of what they actually need to be assigned for production workloads as opposed to giving them all the resources they have available. By being more efficient with assigning memory resources, Hyper-V administrators can be much more efficient and cost effective in how virtual machines are provisioned with resources.
NUMA node considerations with Hyper-V virtual machines
With today’s multiprocessor, dense memory configurations of today’s physical Hyper-V hosts, Hyper-V server exposes the NUMA or Non-Uniform Memory Access presence to its virtual machine guests. The virtual machine guests can then use this information intelligently for resource allocations. With today’s CPU architecture, CPUs have the ability to access the memory locally with the local memory controller and also access the “remote” memory found attached to other CPUs in the hardware configuration. The remote memory access is going to be slower than the local memory access. Generally, this becomes a concern with the wide VMs that are configured with many vCPUs and large quantities of memory. When a virtual machine is configured with more vCPU cores or memory than the physical NUMA node is able to provide, the Hyper-V virtual machine will then span NUMA nodes.
Typically, these kinds of wide configurations are found on NUMA aware application virtual machines such as Microsoft SQL Server virtual machines that benefit from having large quantities of memory. Spanning NUMA nodes with applications that are not NUMA aware can certainly result in performance issues. Wide VMs in a Hyper-V cluster can also cause issues with performance of other virtual machines in the cluster when the wide VM is migrated between cluster hosts.
NUMA configuration and allocation of virtual machines can be at least partially controlled with the Preferred Owners and Possible Owners Hyper-V virtual machine guest settings. These settings allow Hyper-V administrators to control which Hyper-V nodes are preferred as owners of the virtual machine guest compute/memory as well as which Hyper-V nodes may not be used in normal cluster operations for housing certain virtual machines.
You can view the Hyper-V Host NUMA configuration using PowerShell By using the following commandlet:
- Get-VMHostNumaNode
This command returns some interesting information about the NUMA configuration:
- NodeID
- ProcessorsAvailability
- MemoryAvailable
- MemoryTotal
- ComputerName
You can also get the status of the virtual machines on the non-uniform memory access NUMA nodes of a virtual machine host or hosts using the commandlet:
- GetVMHostNumaNodeStatus
Not however when NUMA spanning is enabled on a host you will see the following error:
NUMA spanning is turned on by default but can be controlled at the per Hyper-V host level. To configure NUMA spanning for a particular Hyper-V host or hosts, right click the Hyper-V host in Hyper-V manager and select properties.
By turning off the NUMA spanning feature of the Hyper-V hosts, you will circumvent the NUMA issues that can occur with wide VMs. However, on the other hand, it will result in less efficient Hyper-V density operation. Migration of Hyper-V virtual machines may fail when they would otherwise be successful with the setting enabled.
NUMA can also be controlled at a per-VM level. Looking at the properties of a Hyper-V virtual machine we can configure the NUMA topology at this level also. The Use Hardware Topology button allows querying the current Hyper-V Server settings for the correct values.
Dynamic Memory and Minimum Memory
Another really slick feature of the Hyper-V hypervisor as it relates to virtual machine memory is the dynamic memory setting. The dynamic memory setting allows Hyper-V server to add or remove memory from a VM as needed, on the fly as it were. This assists Hyper-V administrators in an automated way to have correctly sized memory settings for Hyper-V virtual machines. If the Hyper-V administrator has inadvertently configured too much memory to a virtual machine, Hyper-V can take unused memory back to be used for other purposes or virtual machines.
This process allows Hyper-V to have a basically elastic configuration when it comes to virtual machine memory. There may even be hours in the day where certain virtual machines fully utilize the configured memory but during off-peak hours this memory is no longer used. Hyper-V can reclaim this memory once again to be used somewhere else.
The dynamic memory functionality is made possible by a special driver called a balloon driver running in the guest operating system and is able to either request more memory from the Hyper-V host or inform the host that it no longer needs the memory it has been allocated. This interaction between the Hyper-V host and the guest operating system requires the Hyper-V integration services are installed inside the guest.
Starting in Windows Server 2012 Hyper-V you have the ability to configure startup and minimum memory. The premise behind the two settings is you may need more memory assigned to a guest operating system to allow it to boot than it actually needs after the boot process. You can set the startup memory higher than the minimum amount of memory. This allows the virtual machine to boot, but then allows RAM to be reclaimed afterwards.
Concluding Thoughts
Sizing the footprint of virtual machines compared to physical server resources can be challenging when coming from the old school world of sizing server resources. In physical server environments, more RAM than needed was often configured on a physical server to have more than needed. However, in the virtual world, this is not practical since it leads to wasted memory and potential issues with NUMA placement. NUMA is extremely important to consider when sizing VMs and in the number of wide VMs that are actually needed in the environment. Windows Server Hyper-V contains some really great mechanisms for controlling memory sizing and placement for virtual machines. Properly sizing memory, considering NUMA in virtual machine provisioning, and making use of dynamic, startup, and minimum memory settings can help Hyper-V administrators very accurately size virtual machines running in their environments.
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
