When designing out a Storage Spaces Direct or S2D design in Hyper-V, there are certainly many things that need to be considered in the design. Storage Spaces Direct provides an enterprise ready software defined storage solution that is capable of tremendous scale, performance, and manageability within the Hyper-V platform.
Instead of the traditional SAN or NAS device that is shared between multiple Hyper-V nodes, storage spaces direct can provide a much more cost-effective solution that provides additional options in how storage and compute resources are configured. There are a couple of options in relation to the Storage Spaces Direct Hyper-V cluster design that allow organizations to aggregate or separate both the computer and storage resources.
The Storage Spaces Direct models are disaggregated and hyperconverged.
- What are the differences between the two models?
- What particular advantages/disadvantages are present with each model of Storage Spaces Direct design?
Let’s look at both of these in a bit more detail as well as some general Storage Spaces Direct Technologies that are used by both technologies.
Storage Spaces Direct Technologies
Datacenter Bridging (DCB) – These are open standards developed by the IEE. DCB helps to avoid traffic loss for all types of traffic without using TCP as part of the solution. It is important with Storage Spaces Direct to enable DB with RDMA over Converged Ethernet (RoCE). DCB can also enable Priority-based flow control so that latency sensitive storage traffic can be given priority.
Remote Direct Memory Access – This technology allows network adapters to carry data directly in memory without using buffers, CPU, or the operating system. Since several layers of the stack are bypassed, this allows RDMA to have drastically higher throughput and also reduces CPU utilization.
SMB Direct – Utilizing the features provided by RDMA enabled network adapters, SMB direct is able to reduce CPU utilization, latency, and increase throughput. Remote file storage can appear as if it is local storage.
Hyper-V Storage Spaces Direct Disaggregated Deployment
Out of the two different configuration deployments that can be used for Storage Spaces Direct, the disaggregated model is the more complicated of the two configurations.
In the disaggregated model, the configuration requires two clusters instead of just one cluster. One of the two clusters is used for compute and One is used for storage.
Let’s look at the requirements for nodes in the disaggregated model. In the disaggregated deployment, the storage component is built using S2D and scale-out file server (SOFS) to provide an independently scalable storage repository for VMs and apps.
Why would you want to do this?
Virtual machines are configured to store their files on the Scale-Out File Server (SOFS) which is accessed through the network using the SMB3 protocol. This allows for scaling Hyper-V clusters (compute) and SOFS cluster (storage) independently.
Disaggregated Deployment Architecture
Storage Spaces Direct Disaggregated Deployment Model Overview (Image courtesy of Microsoft)
Compute Cluster – Needs at least two nodes (three recommended). Maximum of 64 nodes
- Two 10 GBps NICs with RDMA
- Requires the Hyper-V role and Datacenter bridging and failover clustering features
Storage Cluster – Needs at least two nodes (four recommended). If you use at least four nodes you get additional high availability benefits such as multi-resilient virtual disks. Maximum is 16 nodes.
- Two 10 GBps NICs with RDMA
- Two cache storage devices and capacity storage devices
- For sizing the RAM requirements, each 1TB of cache requires 10 GB of memory
- Requires internal DAS storage or a SAS connected JBOD enclosure (SAS, SATA, or NVMe)
- Storage is accessed using the SMB3 protocol
- Requires installing the Failover Clustering and Datacenter bridging features, as well as the File server role
In System Center Virtual Machine Manager (SCVMM) you can create the disaggregated deployment by setting up SOFS and then enabling S2D.
- Click Fabric Resources > Create > File Server Cluster
- In General Configuration, specify a cluster name, select a host group, and select Storage attached directly to each cluster node (Storage Spaces Direct)
- In Resource Type, specify the RunAs account with local admin permissions on the servers you want to add to the cluster, and specify whether to add existing Windows servers or bare-metal machines
- In Cluster Nodes, define a list of computers to add to the cluster
- On the Summary page, confirm the settings and then click Finish
Typically, in larger environments you will find the disaggregated model of deployment is the most suited for running production workloads.
Licensing Disadvantages
Due to the fact that the disaggregated deployment utilizes both a compute AND a storage cluster, Windows Server licensing costs are more expensive with the disaggregated deployment. However, if you want to disassociate the storage from the compute nodes, the disaggregated deployment is the configuration option you want to go with.
Hyper-V Storage Spaces Direct HyperConverged Deployment
The other deployment model that can be configured with Hyper-V Storage Spaces Direct is the Hyperconverged configuration. In the Hyperconverged deployment, the compute and storage cluster are one in the same. In other words, you don’t have a separate cluster that is managing the compute resources and one managing the storage resources.
The storage devices are contained in the compute nodes themselves. The HyperConverged deployment model is a simpler design that is well suited for smaller deployment scenarios. Virtual machine files are stored on local Cluster Shared Volumes.
Hyperconverged Deployment Architecture
Storage Spaces Direct Hyperconverged Deployment Model (Image courtesy of Microsoft)
Computer/Storage Cluster
- Need at least two nodes (four recommended to take advantage of Multi-Resilient Virtual Disk MRV)
- Identical CPU, RAM, Storage Controllers, Network Adapters, etc
- 128 GB memory required
- Each node must have the same number of storage devices as well as the same number of disks of each type (HDD, SSD, NVMe)
- Two cache and two capacity devices required for storage
- 10 GB of memory is required for every 1 TB of cache
- 10 GBps NICs with RDMA enabled (preferred)
- Hyper-V role, Datacenter Bridging, and Failover clustering must be installed
To enable Storage Spaces Direct:
- Enable S2D on a Hyper-V cluster currently managed in the VMM fabric
- Add an existing Hyper-V cluster to the VMM fabric, with or without S2D enabled
- Provision a Hyper-V cluster and enable S2D from existing Hyper-V hosts
- Provision a Hyper-V cluster with S2D enabled from bare-metal servers
- You can’t currently enable S2D in a hyper-converged deployment on a Hyper-V cluster deployed from bare metal computers with the Nano operating system
Performance Considerations
When looking at the differences in the architecture of both deployment models, are there differences in performance? With the disaggregated model, the storage is not directly attached to the compute cluster as it is in the hyperconverged model, so there is a bit of difference in the total IOPs performance when looking at both deployments head to head.
Microsoft released a performance considerations post that looks at the overall differences in utilizing hyperconverged, disaggregated (SMB) and disaggregated (SMB Direct). The hyperconverged deployment wins in total IOPs, however, the SMB direct deployment seems to be a more consistent deployment when looking at deviation of latency and IOPs.
You can take a look at the full post comparing performance between the deployment types here:
https://blogs.technet.microsoft.com/larryexchange/2015/11/02/performance-consideration-for-disaggregated-and-hyper-converged-configurations/.
Takeaways
Microsoft Windows Server 2016 Hyper-V certainly gives customers many options when it comes to how compute and storage resources are deployed with Storage Spaces Direct. Windows Server 2016 Storage Spaces Direct can be deployed in the Hyperconverged and Disaggregated models of deployment. Each has use cases and benefits.
With the Disaggregated deployment, you are decoupling the compute from storage so each can be scaled independently of one another and is generally the better suited deployment in large scale roll outs.
Hyperconverged deployments are less complex deployments that allow running both compute and storage clusters all in one. This is generally the model that will be used in smaller deployments. It is also desirable from a licensing standpoint compared to the disaggregated model.
Storage Spaces Direct is a great technology that lends itself to some of the cutting-edge datacenter technologies out there today such as data center bridging, RDMA, and SMB Direct. By utilizing these new technologies in Hyper-V Storage Spaces Direct deployments, Microsoft enables customers to have a powerful solution that meets the performance demands and scaling needs of both today as well as tomorrow.
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
