At last, a full (opensource!) GPU virtualization solution that allows virtualized guest operating systems to “own” a shared Intel GPU yet allows applications to run with near-native performance! This means games, both DRM and free media, plus OpenCL applications can  run in separate guest operating systems on a single Intel GPU system (convenient for users) or in a shared cloud environment (a nice feature that can be offered by cloud providers). This is an Intel-only GPU solution. For more about other – non-Intel – GPU virtualization solutions, see our popular TechEnablement article, “MultiOS Gaming CUDA & OpenCL Via a Virtual Machine“.

Full GPU Virtualization on Intel GPUs (image courtesy the Linux foundation)

Performance is reported to be within 95% of utilizing the native GPU in a non-virtualized environment:

XenGT provides near native GPU performance.

More detailed performance and overhead analysis is provided in the paper,  A Full GPU Virtualization Solution with Mediated Pass-Through” by Kun Tian, Yaozu Dong, and David Cowperthwaite of Intel Corporation. To learn more about Intel’s push into the cloud with GPU virtualization, see the September 2014 IDF presentation “Intel® Processor Graphics, Ready to Serve from the Cloud“. They note, “Major ISVs and OEMs are aligning with Intel to productize Intel GVT based solutions. Open source developers might also find Intel GVT portfolio with Intel processor-graphics products equally enticing.”

XenGT was the code name for the Xen implementation of Intel’s vGPU solution, now called GVT-g

The Intel gVirt project is an opensource dual GPL/BSD license project currently based on Xen (hence the codename of XenGT) that has also been marketed by Intel as GVT-g. We are told a  KVM implementation will be forthcoming. Citrix is also bringing support to this technology to their Xen ClientEnterprise product.

Following is a video of the June 2014 Usenix presentation introducing  Intel® GVT-g (then called “gVirt”, but they are one in the same) and the paper, “A Full GPU Virtualization Solution with Mediated Pass-Through“. The authors note that Intel® GVT-g is a product level GPU virtualization implementation advertised as providing:

1. Full GPU virtualization running native graphics driver in guest
2. Mediated pass-through that achieves both good performance and scalability, and also secure isolation among guests.

Video Player

The authors also note that “gVirt [Intel® GVT-g] presents a virtual full-fledged GPU to each VM. VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from guest are trap-and-emulated at minimal cost. Experiments demonstrate that gVirt can achieve up to 95% native performance for GPU intensive workloads, and scale well up to 7 VMs.”

Intel® GVT-g utilizes two key concepts that enable both performance and usability:

1. The Intel® GVT-g stub that extends the Xen vMMU module to selectively trap or pass-through guest access of certain GPU resources. Traditional Xen supports only pass-through or trap of the entire I/O resource of a device. All the trapped accesses are forwarded to the mediator (see sharing discussion below) for emulation. The mediator uses hypercalls to access the physical GPU.
2. Address space ballooning eliminates  virtual graphics address translation overhead through address ballooning by ensuring that the guest OS view of global graphics memory space is exactly the same as (but restricted to a fraction of) the host view.  Succinctly, Intel® GVT-g exposes the partitioning information to the VM graphics driver through the VM_info MMIO window. The graphics driver marks VMs’ regions as ‘ballooned’ and reserves them from its graphics memory allocator. Thus the physical addresses in the guest OS can be directly used by the hardware. Graphics address space ballooning is different from traditional memory ballooning techniques: Memory ballooning is for memory usage control and is focussed on  the number of ballooned pages, while address space ballooning marks special memory address ranges.

Sharing one GPU across multiple virtualized guest operating systems

The combined GVT-g stub and address space ballooning means that Intel® GVT-g can run the native graphics driver inside a VM with direct access to performance-critical resources. Privileged operations are emulated by the mediator (shown in the first image) by trap-and-emulation. The Intel® GVT-g mediator is the manager for all the guest vGPUs and is also responsible for scheduling of vGPU operations. (Currently round-robin scheduling is utilized.)  The mediator  handles the physical GPU interrupts including the generation of a virtual interrupt to the designated VMs.

XenGT Intel GPU Page Table Virtualization (click for larger size, courtesy the Linux foundation)

Security issues are addressed in the gVirt design:

• A VM is prohibited from mapping unauthorized graphics memory pages.
• All the GPU registers and commands programmed by a VM are validated to only contain authorized graphics memory addresses.
• The  GVT-g design addresses denial-of-service attacks, where a VM might deliberately (or mistakenly) trigger lots of GPU hangs.

Virtualized guest operating systems can display DRM and free content

The virtualized guest operating systems can display DRM and other content through a pass-through mechanism:

Intel Render Engine Virtualization (image courtesy the Linux foundation)

Both direct and indirect modes are supported:

Direct-mode Render Virtualization (image courtesy the Linux foundation)

Indirect-mode Display Virtualization (image courtesy the Linux foundation)

Current status

• Initial release in 09/2013, including all major features
  • Multiple VMs (Dom0 + 3 VMs) running simultaneously
  • Display switch among VMs
• Second release in 03/2014
  • More stability improvement
  • Support guest resolution changes
  • Enhanced support for multiple display and hotplug
  • Preliminary support for GPU recovery
  • More developing/tuning work ongoing

Try it out!

• https://github.com/01org/XenGT-Preview-kernel
• https://github.com/01org/XenGT-Preview-xen
• https://github.com/01org/XenGT-Preview-qemu

Following is a video from the Xen project on XenGT.