Run a hypervisor within a hypervisor before upgrading to vSphere 6
Image: Ravello Systems
Enterprise virtualization features such as snapshots and clones allow data center managers to test system upgrades prior to production implementation. For example, these tools take the guesswork out of the impact of moving from one major version of Red Hat to a new version. One capability that has been missing is the ability to test the hypervisor's migration to a new version. Nested virtualization is a potential solution.
A hypervisor within a hypervisor
The basic concept of nested virtualization is the ability to run a hypervisor within a hypervisor. For example, since vSphere 5.1, VMware supports installing one or more ESXi servers within an instance of ESXi.
To help visualize the concept, think of the process of performing a physical to virtual (P2V) conversion. Using a product such as VMware Converter, an administrator can take a physical workload and convert it to run within a hypervisor.
Now take this concept and apply it to a physical hypervisor. If an administrator performs a P2V on the hypervisor, a test platform is created for the hypervisor platform. The capability is made possible by CPU assisted virtualization extensions from Intel and AMD.
Limits to nested virtualization
Running a hypervisor within a hypervisor creates a high level of CPU overhead; therefore, nested virtualization isn't a configuration that is supported on production workloads. Also, there are certain capabilities that can't be tested. For instance, it isn't possible to replicate hardware-specific features such as PCI Flash or Fiber Channel configurations in a nested environment.
In addition, testing for performance issues within a nested environment is problematic. Due to the overhead and the lack of direct access to hardware such as SSDs, the test cases are limited to functionality and compatibility testing. However, the ability to test functionality and compatibility of existing workloads via nested virtualization reduces the overall risks to a migration.
Hardware options
Several options exist for running nested virtualization-based tests. One option is to dedicate a portion of your production capacity to the test. I highly recommend avoiding this strategy unless you are very familiar with the networking in vSphere. There's a high risk of bringing up duplicate workloads within production.
A second option is to leverage a dedicated lab environment that is cut off from your existing network. The lab doesn't have to provide equal the performance capability of your production cluster. As a point of reference, I've been able to run three ESXi hosts with 8 GB of RAM each, a vCenter Server, and a guest Linux VM on a single physical server with 16 GB of RAM. The performance was more than acceptable for a test environment.
A third option is to leverage a cloud solution. VMware vCloud Air is capable of running nested ESXi within its service. Startup Ravello Systems also supports running ESXi within AWS and Google Compute. The previously mentioned lab cost me $1.50/hr within Ravello's AWS option. To better visualize nested virtualization, I created a video of my Ravello lab.
Conclusion
Nested virtualization is a potential peek into the future of data center virtualization. As x86 compute performance increases, nested virtualized environments will become more common for production workloads.
How have you tackled testing the upgrade of you hypervisor from one version to the next? Share your experiences in the comments section.
