VMware Products and Hardware-Assisted Virtualization (VT-x/AMD-V)

Introduction

Both major x86 chip vendors now provide hardware-assisted virtualization (HV) support. Intel's x86 hardware virtualization technology is known as VT-x or, more commonly, simply VT (Virtualization Technology). Previously, it was code-named "Vanderpool." AMD's x86 virtualization technology is known as AMD-V. Previously, it was known as "SVM" (Secure Virtual Machine), and before that, it was code-named "Pacifica." These competing technologies are remarkably similar under the hood, particularly in the first generation. Performance of the first generation hardware was somewhat lackluster, and for that reason, VMware products still default where possible to binary translation without hardware assistance on older HV-capable hardware.

Both vendors have made incremental improvements to their hardware-assisted virtualization technology, and now both vendors offer a major advancement in the form of virtual MMU support, known as nested paging or second level address translation (SLAT). AMD was the first to come out with nested paging (SLAT) in their Family 10H (3rd Generation Opteron) processors, and they dub their nested paging (SLAT) technology Rapid Virtualization Indexing (RVI). Intel introduced a comparable feature in their Core i7 processors, and they dub their nested paging (SLAT) technology Extended Page Tables (EPT). With nested paging (SLAT) support, along with improvements in VM-entry/VM-exit latencies, hardware-assisted virtualization now outperforms binary translation in most situations. However, TLB misses are much more expensive in a nested paging (SLAT) environment, so workloads that over-subscribe the TLB are potentially still good candidates for binary translation without hardware assistance. 32-bit Windows XP is also a good candidate for binary translation, due to its frequent TPR accesses.

As the hardware has evolved, VMware software support of hardware-assisted virtualization has also evolved. The following sections detail the hardware-virtualization support in some of our products.

Workstation 5.5 and ESX 3.0

Intel VT-x support was first introduced in Workstation 5.5 to support 64-bit guests on Intel hardware. Since Intel does not provide segment limit checks in 64-bit mode, VT-x is a requirement to provide isolation between 64-bit guests and the virtual machine monitor (or hypervisor). Though unsupported, it is also possible to run a 32-bit guest using VT-x. The configuration option for running a 32-bit guest with VT-x is:

monitor_control.vt32 = TRUE

There is no support for EPT or AMD-V in these products.

Workstation 6.0 and Fusion 1.0

Workstation 6.0 has the same VT-x support as Workstation 5.5. Experimental support was added for AMD-V and RVI. The configuration option for running a guest (either 32-bit or 64-bit) with AMD-V is:

monitor_control.enable_svm = TRUE

RVI will be used by default on RVI-capable hardware. Note that this is the only VMware product that offers support for AMD-V on Family 0FH processors (2nd Generation Opteron) with AMD-V support. This support is experimental.

ESX 3.5

ESX 3.5 only supports VT-x for running 64-bit guests. The experimental 32-bit VT-x support was eliminated. ESX 3.5 also provides the first official AMD-V support, but only for chips with RVI. With this release, we dropped the monitor_control flags for hardware-virtualization. The configuration option for running a guest (either 32-bit or 64-bit) with AMD-V and RVI is:

monitor.virtual_mmu = hardware

Unlike the boolean monitor_control flags used previously, the monitor.virtual_mmu option has three settings: automatic, software and hardware. Selecting RVI implicitly selects AMD-V, since RVI cannot be used with binary translation.

Workstation 6.5 and Later, Server 2.x, ESX(i) 4.0 and Later, Fusion 2.0 and Later

These products add official support for VT-x and AMD-V for all guests, both 32-bit and 64-bit. EPT support is introduced for EPT-capable hardware. VT-x or AMD-V can be used in conjunction with a software MMU or with nested paging (SLAT) on hardware that supports it. Note, however, that hardware-assisted virtualization is only supported on 64-bit hardware, and there is no support for AMD-V on Family 0FH processors (2nd Generation Opteron).

The configuration option for selecting binary translation is:

monitor.virtual_exec = software

The configuration option for selecting VT-x/AMD-V is:

monitor.virtual_exec = hardware

The configuration option for selecting the software MMU (aka shadow paging) is:

monitor.virtual_mmu = software

The configuration option for selecting nested paging (SLAT) is:

monitor.virtual_mmu = hardware

Note that you cannot mix binary translation with nested paging (SLAT).

Workstation 6.5 (and later) and Fusion 3.0 (and later) allow you to make these selections through the UI, in the virtual hardware "Processors" configuration. ESX(i) 4.0 (and later) provides a configuration selection through the vSphere client. For the other products, you have to edit the VM configuration files by hand.

Whether you edit the configuration file by hand or use the UI, you should be careful to select an execution mode that is supported by your hardware. In particular, if you select nested paging (SLAT) on hardware without nested paging (SLAT) support, the execution mode may revert to binary translation. If you select binary translation for a 64-bit guest on Intel hardware, the execution mode will dynamically switch to VT-x as soon as the guest enters long mode (typically partway through the guest boot process).

If you do not select a particular execution mode, a default mode will be chosen for you. The defaults are incredibly complex, depending on your hardware capabilities and the guest OS type. In general, the default mode is likely to be the best (highest performing) mode for your hardware and guest OS, but the heuristics are not infallible. You should feel free to benchmark your workload using the execution modes available on your hardware and choose what works best for you.

Note that some products default to binary translation for 32-bit Windows 2003, due to the frequent TPR accesses of that guest. However, Service Pack 2 addresses this issue. If your guest is 32-bit Windows 2003 and you have installed Service Pack 2, you should change the execution mode to hardware-assisted virtualization on supported hardware for the best performance.

For more information, please see http://www.vmware.com/resources/techresources/10036.