Beginner’s Guide for Microsoft Hyper-V: What is Non-Uniform Memory Access (NUMA) – Part 16

Introduction

A crucial aspect of any virtualization solution is memory management. In addition to scheduling CPU operations, hypervisors need to manage and juggle access to the physical host memory from the various virtual machines requesting access to those resources. Hyper-V supports NUMA (Non-Uniform Memory Access). In this article, we will look at, What is NUMA, and how is it configured in Hyper-V? What do admins need to consider when configuring Hyper-V virtual machines spanning NUMA nodes?

Table of Contents

1. Non-Uniform Memory Access (NUMA)
2. NUMA-aware applications
3. Hyper-V NUMA Spanning
4. Hyper-V Virtual NUMA
5. Dynamic Memory and Virtual NUMA
6. Wrapping Up

Non-Uniform Memory Access (NUMA)

Today, the predominant CPU/memory architecture with multiprocessor system designs is Non-Uniform Memory Access (NUMA). Most servers used today have a dual CPU architecture with directly attached memory via a memory controller. This grouping of resources is referred to as a NUMA node.

From the perspective of each CPU, the memory directly attached to the CPU package and its memory controller is viewed as local memory, and the memory connected to the opposite CPU is remote.

There is no performance penalty when resources are accessed from the local NUMA node. However, there will be a performance penalty in terms of additional latency accessing the remote memory found in the remote NUMA node.

NUMA-aware applications

Modern operating systems and many enterprise applications have been written and optimized in such a way to minimize the impact of NUMA remote memory access. For example, Microsoft SQL Server is an application that adapts to NUMA configurations. It attempts to allocate storage for data and schedule CPU threads to satisfy these in the same NUMA node if possible.

Even if you are not running a NUMA-aware application inside a virtual machine, modern operating systems can take advantage of NUMA optimizations and benefit from the virtual NUMA topology presented to the VM from Hyper-V.

Hyper-V NUMA Spanning

When running virtual machines, a virtual machine will typically have resources that fit within one NUMA node or another and will achieve the best performance by accessing resources local to the CPU package it has been assigned, including memory access.

To achieve the best performance for the virtual machine, Hyper-V attempts to allocate all the memory to that virtual machine from a single physical NUMA node. This configuration is ideal as it allows the virtual machine to access memory local to the physical CPU resources it runs.

You can see this configuration when you look at the Hyper-V host settings in Hyper-V Manager. By default, under the NUMA Spanning configuration, you will see the configuration setting Allow virtual machines to span physical NUMA nodes. As described, spanning NUMA nodes can help you run more virtual machines simultaneously. It also allows for providing more memory to workloads than what is available in a single NUMA node.

As a note. Any Hyper-V virtual machine created on the Hyper-V host must fit within a single NUMA node if you disable NUMA spanning. If you create a virtual machine requiring more resources than can be satisfied within a single NUMA node and NUMA spanning is disabled, the virtual machine will not start.

Hyper-V Virtual NUMA

However, organizations may have virtual machines that are so-called “wide virtual machines” or virtual machines that may have memory or CPU configurations that extend beyond what can be provided by a single NUMA node configuration.

For example, if a NUMA node contains a 20-core processor and 512 GB of memory, and a virtual machine has been configured with 12 CPU cores and 768 GB of memory, the memory configuration is beyond what can be provided using a single NUMA node.

With these large virtual machines, it will utilize multiple NUMA nodes to satisfy the resource requirements for the configured virtual machine. However, suppose the virtual machine is not aware of the physical NUMA node configuration of the Hyper-V host, in such cases, it will not take advantage of the NUMA node optimizations, and performance will not be optimal.

Beginning with Windows Server 2012, Microsoft introduced the ability to present virtual NUMA topology to Hyper-V VMs. The default configuration of the virtual NUMA topology has the configuration match what is configured for the physical NUMA topology of the Hyper-V host.

There are tremendous advantages in exposing the virtual NUMA topology to the Hyper-V host. First, it lets the virtual machine know the most efficient access to local and remote NUMA resources. Second, it enables NUMA-aware applications like Microsoft SQL Server to take advantage of NUMA optimizations.

There is no distinction between physical and virtual NUMA node constructs for a virtual machine. If possible, memory inside the virtual machine is allocated efficiently for fast local memory access, avoiding the performance penalties of remote memory access.

When you look at the settings for a Hyper-V virtual machine and expand the Processor settings, you will see the NUMA configuration. In the NUMA configuration, you can set:

• Maximum number of processors
• Maximum amount of memory (MB)
• Maximum number of nodes allowed on a single socket
• Number of hardware threads per core
• Use hardware Topology

If a Hyper-V virtual machine is moved to a Hyper-V host with a different NUMA configuration, you can restore the default virtual NUMA topology using the Use Hardware Topology button on the NUMA configuration page for the virtual machine.

Dynamic Memory and Virtual NUMA

There is a caveat to NUMA spanning when using a virtual machine with Dynamic Memory enabled. Both cannot be used at the same time. When Dynamic Memory is enabled on a virtual machine, it has access to only one virtual NUMA node, and no virtual NUMA topology is presented to the virtual machine, regardless of the NUMA settings.

Wrapping Up

Hyper-V provides many optimizations to take advantage of the modern Non-Uniform Memory Access (NUMA) architecture, providing the most efficient access to resources in a Hyper-V host. The Hyper-V NUMA spanning configuration and virtual NUMA allows Hyper-V virtual machines to span across multiple NUMA nodes and take advantage of NUMA optimizations for NUMA-aware applications and operating systems. Understanding the basics of NUMA is an excellent way for beginners to know how Hyper-V handles memory access and virtual hardware configurations for VMs running in a Hyper-V environment.

Related Posts:
Beginner’s Guide for Microsoft Hyper-V: Hyper-V Dynamic Memory – Part 17
Hyper-V Virtual Machine Memory Considerations
Hyper-V Mastery: A Step-by-Step Guide for Beginners to Elevate Your IT Skills and Boost Your Career
What is VMware Hot Add & Hot Plug? How will it impact NUMA?

Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.