There are many great things you can do on top of today’s modern hypervisor systems that provide a great way to solve business problems and technical challenges. One of the really neat things you can do with today’s hypervisors such as VMware vSphere is, you can configure nested virtualization. This opens up many different possibilities of testing and running hypervisors inside of VMware vSphere.
Some key advantages are:
- Testing and Development: Nested virtualization allows you to set up and test Hyper-V virtual machines within your existing VMware ESXi environment. This is particularly useful for developers or IT professionals who need to work with both Hyper-V and VMware technologies.
- Resource Isolation: By running nested Hyper-V virtual machines in VMware ESXi, you can isolate the resources used by each VM, ensuring that the performance of the parent VMware ESXi host and other virtual machines remains unaffected during testing or experimentation.
- Cost and Space Savings: With nested virtualization, you can consolidate multiple virtualized environments onto a single physical server, reducing the need for additional hardware. This leads to cost savings and more efficient use of available server resources.
Overall, nested Hyper-V virtualization in VMware ESXi offers a flexible and efficient way to work with multiple virtualization technologies, making it an invaluable tool for testing, development, and resource optimization.
In this post, we will take a look at configuring nested Hyper-V VMs on a VMware ESXi Server and see how this can be configured for use in a lab or other environment.
Table of Contents
- What is Nested Virtualization
- Why Run Hyper-V Inside of VMware?
- Lab Environments
- No Additional Hardware or Network Gear is Needed
- Easily Provision and Tear Down Hyper-V Hosts
- Requirements for Nested Hyper-V VMs on a VMware ESXi Server
- Exposing Hardware-Assisted Virtualization to the Guest OS
- Enabling Promiscuous Mode and MAC Address Forged Transmits
- Can You Backup Nested Environments?
- Wrapping Up
What is Nested Virtualization
Before diving into how to configure VMware vSphere to enable nested virtualization of Hyper-V inside of ESXi server, let’s take a look at what nested virtualization is exactly. When thinking about nested virtualization, we are talking about running a hypervisor inside of a hypervisor. This means you can run Type 1 hypervisors like VMware vSphere ESXi and Hyper-V inside of a VM on top of, in this case, VMware vSphere ESXi.
The hypervisor running in the VM on top of VMware vSphere ESXi server simply views the VM hardware as its physical host hardware on which it can run guest virtual machines. This allows some very interesting scenarios when using your VMware vSphere environment to host other hypervisors like Microsoft Hyper-V.
An important point to make in regards to VMware vSphere is nested virtualization of other hypervisors like Hyper-V inside a vSphere VM is not a supported scenario. It’s not just other hypervisors outside of VMware vSphere. Even running ESXi nested inside of a physical ESXi host environment is not supported. This is officially noted by VMware in the KB article Support for running ESXi/ESX as a nested virtualization solution (2009916).
The following configurations are technically possible when running VMware ESXi inside of other VMware hypervisor products but is not supported. This includes the following.
- VMware ESXi/ESX running in VMware Workstation or VMware Fusion
- VMware ESXi/ESX running in VMware ESXi/ESX
- VMware ESXi/ESX running in other third-party hypervisor solutions
Of course, running Hyper-V falls under this same type of support disclaimer. There is only one situation where nested virtualization is supported and that is the nested vSAN Witness appliance.
Outside of this supported use case with the vSAN Witness Appliance, nested virtualization is not supported for the production environment. In other words, if you run into issues using nested virtualization, you are on your own with support.
As long as you understand the support scenario with nested virtualization, it is a great feature to leverage for other purposes. It can be used very effectively for lab environments, dev, test, and other similar types of scenarios.
Why Run Hyper-V Inside of VMware?
Why would you want to run a hypervisor inside another hypervisor? More specifically, why would you want to run Hyper-V inside of a VMware ESXi VM? As already mentioned, this is not a supported configuration for production use. Running Hyper-V inside of VMware provides many great advantages. Let’s look at the following:
- Lab environments
- No additional hardware or network gear required
- Easily provision and tear down Hyper-V hosts
Lab Environments
This is arguably the most common use case for provisioning a Hyper-V host inside a VMware virtual machine. Nested virtualization allows provisioning lab environments, dev, test, and other possible use cases very easily. Hyper-V labs may be used for:
- Learning
- Software testing
- POC’ing Hyper-V for various parts of your infrastructure or as a replacement
- Mimicking a Hyper-V environment in another part of your infrastructure
- Patch testing
In many small to medium-sized environments, there may be only the availability of a VMware vSphere environment and no Hyper-V hosts. If administrators want to get their hands on a Hyper-V host to learn and develop their skills outside of the VMware ESXi hypervisor, the nested virtualization capabilities found in VMware vSphere provides the perfect way to provision a Hyper-V host inside a VMware vSphere environment.
No Additional Hardware or Network Gear is Needed
With this capability of nesting Hyper-V inside of VMware vSphere, there is no need for finding additional hardware for a lab environment or POC of sorts. This can all be provisioned inside of vSphere. The robust virtual network capabilities found inside of vSphere also allow provisioning all of the specialized networks you will need if you decide to provision a Hyper-V cluster. This may include Live Migration, Storage, and Cluster networks to name just a few. Using virtual switch port groups, you can easily simulate the configurations that would be accomplished with physical network gear otherwise.
Easily Provision and Tear Down Hyper-V Hosts
This advantage is not specific to nested virtualization but rather VMware vSphere virtualization in general. VMware has very rich automation tool sets including PowerCLI. This allows easily spinning up and destroying VMs as you need them. Entire lab constructs can be automated for provisioning and deprovisioning. This serves to make nested virtualization even more useful with the easy capabilities for automating your infrastructure.
Requirements for Nested Hyper-V VMs on a VMware ESXi Server
Once you have decided to make use of nested virtualization, can you simply just create a new VM in vSphere ESXi and start loading up your Hyper-V VM? Well, not exactly, however, the process is still very easy to use when you think about the enormous complexity required underneath the “hood” so that nested virtualization actually works. There are really two main areas to consider with the requirements for running nested Hyper-V VMs on a VMware ESXi Server. This includes:
- Exposing hardware-assisted virtualization to the guest OS
- Enabling MAC address forged transmits
Let’s take a look at each of these and their importance to running nested Hyper-V VMs on a VMware ESXi Server.
Exposing Hardware-Assisted Virtualization to the Guest OS
This first requirement you will find to be absolutely required. It is a setting underneath the settings for the individual VM that you are using to house your Hyper-V installation. Edit Settings on your nested Hyper-V VM, expand CPU and place a checkbox next to the Expose hardware-assisted virtualization to the guest OS.
What happens if you do not enable this flag for hardware-assisted virtualization in VMware for your Hyper-V VM? You won’t see any error simply installing your Windows Server operating system as this will work as installing any other VM. However, when you get to the point of installing the Hyper-V role, you will see an error if this setting is not enabled for the virtual machine. Notice the error below, “Hyper-V cannot be installed: The processor does not have required virtualization capabilities”.
As a side note, this flag needs to be set for either Hyper-V Core installations or with the Hyper-V role installed in Windows Server with the Desktop Experience installed. This capability is required in any nested installation of the Hyper-V role inside VMware vSphere.
You can also set this flag with a bit of PowerCLI code which allows easily setting the flag on the VM specified.
$vmName = ‘MyHyperVVM’
$vm = Get-VM -Name $vmName
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.nestedHVEnabled = $true
$vm.ExtensionData.ReconfigVM($spec)
Enabling Promiscuous Mode and MAC Address Forged Transmits
Most likely, if you are configuring a Hyper-V nested installation inside of VMware vSphere, you will want to be able to not only install a server with the Hyper-V role but also run a nested Hyper-V VM on top of the nested Hyper-V server that you are running.
You may not care about having the ability for network connectivity to the Hyper-V VM, however, you may want to be able to establish real network connectivity to the nested virtual machine running on top of the nested Hyper-V host in the VMware virtual machine.
If you want to have this functionality, there are a couple of network configuration settings that you may need to make on your VMware vSwitch that your nested Hyper-V host is connected to.
You may need to do this since there is a new advancement in VMware vSwitch technology and nested virtualization since the release of VMware vSphere 6.7. Prior to the release of VMware vSphere ESXi 6.7, VMware did not implement MAC learning on a vSwitch like it is found in a real physical switch. This was true of either the vSphere Standard Switch or the vSphere Distributed Switch.
Since this is the case with the VMware vSwitch prior to vSphere 6.7, when a virtual switch receives packets that do not have a MAC address matching the nested hypervisor hosts vmnic’s pNIC MAC address (in the case of a nested ESXi host), the packets will be dropped.
To get around this limitation, there are two settings that need to be enabled on the vSwitch, promiscuous mode and forged transmits.
What about vSphere 6.7 and higher? Before the release of vSphere ESX 6.7, VMware started doing work in the area of nested virtualization and MAC learning functionality in relation to virtual switches. A “MAC Learning Fling” was released that introduced the MAC learning functionality so that you wouldn’t have to enable promiscuous mode and forged transmit settings.
This setting was implemented with the release of VMware vSphere 6.7 ESXi.
What are the requirements of the new MAC Learning feature in ESXi 6.7?
There are a few requirements that you need to have in place before you can take advantage of the new MAC learning functionality in vSphere 6.7. They are as follows:
- Upgrade vCenter Server and ESXi servers to vSphere 6.7
- Run the vSphere Distributed Switch (vDS)
- Upgrade the vDS to the latest version (6.6)
- Managed per vSphere Distributed Switch Port Group basis
- Currently managed with the vSphere API
One thing to note, the new MAC Learning functionality with vSphere 6.7 and a vDS 6.6 vSwitch is not enabled by default. You will have to set the flags appropriately with the API.
To make this process much easier, William Lam, a Staff Solution Architecture has written a couple of PowerCLI functions that make checking the state of MAC Learning and setting MAC Learning in your vSphere environment extremely easy. Download William’s PowerCLI script from his Github repository here:
https://github.com/lamw/vghetto-scripts/blob/master/powershell/MacLearn.ps1
An example of the output of the Get-MacLearn function checking a vSphere Distributed Switch Port Group looks like the below. Note the fields:
- MacLearning
- NewAllowPrimiscuous
- NewForged Transmits
- NewMacChanges
- Limit
- LimitPolicy
This is what a vDS 6.7 vSwitch port group will look like with default settings.
For nested ESXi installations, and by extension, Hyper-V VMs running inside your vSphere ESXi hypervisor, you want to set the settings to the following:
- MAC Learning: true
- Promiscuous mode: False
- Forged Transmit: True
- MAC Changes: False
- Limit: 4096 (you can set this or leave it the default value)
- Limit Policy: Drop (you can set this or leave it the default value)
The above recommendations can be set with the following PowerCLI code:
- Set-MacLearn -DVPortgroupName @(“DPG-Servers”) -EnableMacLearn $true -EnablePromiscuous $false -EnableForgedTransmit $true -EnableMacChange $false
As we now expect, as you notice above, the Promiscuous mode setting is configured to False. Even with MAC Learning enabled, the Forged Transmit is still set to True.
The MAC Learning functionality contained in vSphere 6.7 allows avoiding some of the security implications of running a nested VM inside your vSphere 6.7 environment by preventing the need for promiscuous mode to be enabled.
Can You Backup Nested Environments?
Even though running nested virtualization is only supported by VMware vSphere for the vSAN Witness appliance, if you are running nested Hyper-V VMs inside of VMware vSphere ESXi, you may want to backup your nested environment. Can this be done?
Yes, absolutely.
Using a modern backup solution, like Vembu BDR Suite, allows you to backup not only your production VMware vSphere or Microsoft Hyper-V environment, but also any nested installations of ESXi or Hyper-V that you may have in the environment.
Wrapping Up
Nested Hyper-V VMs on a VMware ESXi Server provides you with the ability to play around with Hyper-V, even if you only have a VMware vSphere environment. As shown, there are many great use cases for nested virtualization. This includes learning, software testing, POC’ing, and patching your Hyper-V hosts.
As powerful as nested virtualization is in VMware vSphere, you might expect the process to enable it to be extremely difficult. However, it only requires a couple of considerations on your part when provisioning a VMware vSphere VM that will contain the Hyper-V installation. This includes exposing the hardware-assisted virtualization flag on the virtual CPU of the Hyper-V VM and also implementing promiscuous mode or MAC Learning for connecting any nested Hyper-V VMs to the network.
All in all, nested Hyper-V VMs on a VMware ESXi server is an extremely powerful capability and allows you to test out and learn Hyper-V without the physical equipment for running the Hyper-V hypervisor.
Download BDRSuite now to secure your Hyper-V environment effortlessly and gain peace of mind.
Learn more about Hyper-V backup with BDRSuite and experience its effectiveness firsthand!
Related Posts:
Nested Virtualization on Hyper-V – An Overview
Beginners’ Guide for Microsoft Hyper-V: What is Nested Virtualization in Hyper-V – Part 45
Beginners’ Guide for Microsoft Hyper-V: Hyper-V Nested Virtualization in Microsoft Azure – Part 51
Installing Hyper-V in Azure Nested Virtualization
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
