Table of Contents
- vSAN stretched clusters
- What problem needed to be solved?
- What is VSAN Stretched cluster failure resiliency?
- Wrap up
The VMware ecosystem is populated by a plethora of products that address all areas of the datacenter. It is without a doubt that those that don’t exist yet will also be addressed in the future like Kubernetes was with the Tanzu portfolio for instance. A foundational part of every existing datacenter is the storage, which is virtualized by VMware vSAN
This product needs no introduction as it has been around since vSphere 5.5. If you were already in the VMware space back then, you’ll know that the first versions were a bit rough around the edges but the product received constant improvements throughout the versions and is now a polished, well-oiled machine. An incredible number of new features made their way in VSAN to both improve user experience and address new use cases such as supporting cloud native storage.
In this article, we are going to look into an improvement that was brought by vSAN 7.0 Update 3. There is a fine line in semantics here as to whether we should call this an improvement or a feature, but you can be the judge of that in the comments.
As a side note, if you remember last year, the release of vSphere 7.0 Update 3 was a bit chaotic as it introduced a bunch of bugs that caused the purple screen to customers which drove VMware to pull the release from distribution, only for it to be fixed and offered again a few months later in December of 2021. The version is now stable and is safe to install.
vSAN stretched clusters
Before we dive into what vSAN Stretched cluster failure resiliency is, let’s start with why this came to be and this calls for a bit of background on vSAN stretched clusters. So, how do these work then?
In a traditional “non-stretched” vSAN cluster, the participating nodes are all part of the same failure domain. Virtual machines are made of components that are scattered across hosts according to their storage policy along with a witness component which serves as a tie breaker to establish quorum in order for the host to understand if it is isolated or not. Note that this concept applies to most clustering models in some shape or form.
The same also applies at the cluster level when distributing the nodes in different sites where each site is configured as a vSAN Fault Domain. vSAN Stretched Cluster is a configuration for environments where disaster and downtime avoidance are requirements. The storage is stretched across both sites so virtual machines can be restarted automatically using vSphere HA instead of executing a lengthy disaster recovery plan (provided networking is taken care of). VMware vSAN Stretched Clusters requires a Witness Host for clusters with 2 active sites with the same number of ESXi hosts distributed evenly between those.
The third site hosts the vSAN Witness Host, a lightweight virtual appliance running nested ESXi, which is connected to both participating sites and supports low bandwidth / high latency links as opposed to the link between the computing sites.
A VM created on a vSAN Stretched Cluster has a copy in each site while all the witness components will be placed on the witness host (fault domains). In case of a site outage, the voting majority is maintained so the resources remain available since the witness and a data component are still around. The VM keeps running or vSphere HA restarts it if it was running on the failed site.
However, if the majority is lost among the components (say the witness is out for whatever reason), the VMs will become (or remain) unavailable since there is no longer a quorum.
What problem needed to be solved?
Now you probably see where I am going with this. Let’s say an organization has a stretched cluster and needs to do some work on one of the sites which requires to power it off, we call this planned maintenance. You happily migrate all your workloads over to the other site and then switch it off. At which point you still hold quorum so no harms done, and everything keeps running.
Now, if for any given reason something were to happen to the witness and it went offline, all the virtual machines would become unavailable as there would be a minority of the quorum left, a less than suitable scenario if you ask the sysadmin in me.
This could be due to a number of reasons like flaky connection, an outage in the third site, routing issues, you name it. I’ll agree that it would be bad luck for something like this to happen and some will even say it is unlikely. To which I would tend to agree on principle but no Sysadmin in their right mind would leave anything so critical to chance when it comes to infrastructure availability.
What is VSAN Stretched cluster failure resiliency?
Because having the witness be a single point of failure when a site is offline (for planned maintenance or not) is less than ideal, VMware tweaked the voting mechanism of VSAN stretched clusters to cover these edge cases and account for witness host failures while a site is already offline.
This quorum mechanism is based on the concept of votes. Each data component holds 2 votes while the witness holds 3. Now after that, the witness would hold more votes than the remaining data site. Meaning it would take it down with it if it were to fail afterwards.
In order to mitigate this, whenever a site goes down, VSAN will rig the voting mechanism by giving 1 vote to the witness components instead of 3 and 3 votes to the remaining data components instead of 1. Meaning that a failure of the witness will not incur a loss of quorum as the data components hold more votes and the VMs can keep running.
Wrap up
VMware VSAN is a very interesting and technical topic. While it feels like it’s been around forever and it may appear simple because it is easy to manage from the vSphere Client, the reality of it is otherwise. Even 8 years later, what happens under the hood keeps getting improved with better stability and availability.
Diving into VSAN Stretched cluster failure resiliency in VSAN 7.0 update 3 was good fun. It is always insightful to get an understanding of the inner workings of products like VSAN.
Even with VMware vSAN’s resilience and availability features, you still need to use efficient backups to protect your production workloads. Even with the software-defined storage and high-availability solutions, data loss can still occur. Vembu BDRSuite provides a great way to protect your virtual machines running in VMware vSAN.
BDRSuite is available for free download. Try it out with your VMware vSAN virtual machines right away.
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
