When it comes to development testing, most are familiar with the concepts of a development and staging environment. The normal lifecycle of code that is well tested will flow from a development test environment to a staging environment and then once tested in staging, will flow into production. With infrastructure changes however, many have long had the practice of simply implementing patches and other major infrastructure changes directly into production. A growing number of enterprise IT environments are becoming 24×7 environments with an online presence that is always on. Infrastructure and operations engineers are having to rethink the way they implement changes into production and start looking at operations more like code. The new movement toward a DevOps way of handling infrastructure is driving many of the changes we are seeing in the processes and procedures of today’s infrastructure engineers. Why are lab environments an important concept now for even infrastructure? How should we as operations engineers be implementing changes into production?
What should a lab environment represent?
To begin with, when we talk about a lab environment for infrastructure, what should the lab environment include? To a reasonable degree, an infrastructure lab environment should represent the infrastructure that is found in a production environment for the particular infrastructure you are testing. For the most part, the infrastructure lab would include the Microsoft Active Directory domain environment that you would find in production as well as the Windows, Linux, and other servers that may be included therein.
Keeping a lab environment up to date is one of the main challenges with making sure the lab correctly represents the production environment. For instance, if we were testing a round of Windows patches in a lab, if the current state of the servers in the lab does not have the same patch levels installed as production, simply testing a new round of patches on this “version” of our lab environment would not fully test what you might see when installing the same patches in production. Certain patches may interact differently with the currently installed patches.
Additionally, it is nice to see a lab environment correctly represent the IP address space that production occupies as well. This requires a bit of network trickery using either virtual routers or physical routers of some sort and creative NAT’ing which allows the same address space to be correctly used in the lab environment and be accessible from the outside production network. The benefit of doing this is that we are correctly replicating our production environment down to the network layer. We want to remember that our point in a lab environment for infrastructure is to as closely as possible mimic what we see in production. The fewer variables that we can introduce into the environment itself before we test changes, the easier it will be to isolate any issues in the lab environment that may have been introduced by whatever it is that we are testing in the lab that is to be rolled into production. What tools can we leverage in the lab to facilitate testing infrastructure changes?
Tools and Ideas to think about
Generally, when we think about snapshots on virtual machines, we do not want to have running in production. However, in a lab environment, snapshots can play a very useful roll in “rinse and repeat” testing. Let’s say you are testing a process of an upgrade of something you want to roll out in production. What if you botch the upgrade process on a SQL Server? If we use a snapshot on the virtual machine that we are testing on in our lab environment, we can then simply roll back to the snapshot that was created prior to the beginning of testing.
Using VMware PowerCLI is a great way to automate both the creation and rollback of snapshots in a VMware lab environment. First, create a folder for the VMs on which you want to create the snapshot. Then we can use a quick PowerCLI one liner to create the snapshot like so:
- get-vm -location “Test VMs” | new-snapshot -Memory -quiesce -name “Before upgrade testing”
Once we are ready to roll back to that particular snapshot we can issue the following command:
- get-vm -location “Test VMs” | set-vm -snapshot “Before upgrade testing” -confirm:$false
Be advised that if you are doing this for multiple VMs, you can take a serious I/O hit on the host DAS or storage array housing these VMs.
Connectivity
We touched on this point above but let’s think about this a little further. If we want to have connectivity from the outside (production) network, we will need to have either a multihomed VM or terminal server that is sitting on a lab VLAN and also in production and then NAT the “lab” interface IP over to the lab network that is running production network address space so that we do not have overlapping subnets. A virtual router with a small footprint can be used to NAT this traffic or perform a netmap of the destination network to the new destination network of the lab subnet which mirrors production.
Depending on the needs of the environment, you would need to decide whether the outside lab subnet would be visible to everyone on your production network or they would need to “double hop” using a terminal server or another VM that has connectivity to the lab network.
Repurposing hardware for a lab
Often, when we thinking about old hardware that we are replacing, most of us don’t think about the possibility of using a production environment/hardware that was recently replaced as a lab environment. Generally speaking this is perfectly acceptable to most as long as they are not testing performance or some other metric related to the performance of the hardware in a lab environment.
A rotation scheme that seems to work quite well when thinking about repurposing hardware for labs especially when a DR facility or environment is in the mix is the following – Production hardware is rotated into DR and the DR hardware is rotated into the lab environment. However, if there is no DR environment, the production hardware would simply be rotated into a lab environment.
In my experience and opinion, having an infrastructure lab is not a “nice to have”, it is a requirement in today’s environments. Thinking about this sort of rotation of our hardware resources makes good use of hardware that might otherwise be disposed of or not utilized fully.
Thoughts
Infrastructure labs are an essential part of today’s multi-faceted enterprise environments that have ever stringent requirements of production uptime. Testing upgrades, patches, infrastructure changes, and other objectives should always be carried out in a lab environment first before rolling into production. In today’s world of ever changing operations requirements, infrastructure engineers are having to work and think more like developers in making changes to their environments, testing those changes first to validate the integrity of systems and applications as a whole.
