Xen runs the guest operating system in a lower privilege ring than the hypervisor and, on paravirtualised kernels, replaces privileged instructions with explicit calls (hypercall) into the monitor. Every honest discussion of virtualisation in a data centre starts there: the gain depends on how the hypervisor traps or avoids privileged operations, and on how much that trapping costs.

These notes were written while I was preparing a technical seminar on Xen for staff from public administrations and from companies in the sector. I try to separate what is measurable and documented from what, in those settings, is taken for granted.

Context and problem

A mid-sized authority’s data centre usually keeps one physical server per service: the file server, the mail server, the line-of-business application, the institutional website, the domain controller. Each uses a fraction of a CPU almost all the time, yet each occupies a slot, a power supply, a network port, a line in the maintenance contract. The practical question is how many of those workloads can be stacked on the same machine without disturbing each other, and what breaks when the hardware underneath gives way.

Virtualisation solves both with the same mechanism: it detaches the operating-system image from the iron it runs on. Xen does this with a type-1 hypervisor, loaded before the operating system, hosting one privileged domain — the control domain, dom0 — plus the unprivileged guest domains, domU.

Architecture

Xen began as a research project at the University of Cambridge; the reference description is the paper presented at SOSP 2003, Xen and the Art of Virtualization. Two techniques coexist in the same hypervisor, and it is worth keeping them apart.

Paravirtualisation requires a modified guest kernel, aware that it runs under a hypervisor. The kernel does not execute privileged instructions and hope the processor handles them: it calls the hypervisor with a hypercall. For memory, the page tables stay readable by the guest, but updates go through the monitor, which validates them. The cost is that an adapted kernel is needed; the benefit is that trap-and-emulate is avoided on the most frequent operations. On Linux this adaptation is no longer a separate patch tree: the paravirt_ops infrastructure has been in mainline since kernel 2.6.23 (2007), and from there a stock kernel can act as a Xen domU.

The other technique is HVM, hardware virtual machine, available since Xen 3.0. It uses the processor’s virtualisation extensions — Intel VT-x, AMD-V — to run an unmodified guest operating system, Windows included. Here device emulation (disk, network, display) is handed to a QEMU-derived process, qemu-dm, running in dom0. It is the only route when the guest cannot be recompiled, and it is also the heaviest on I/O: every access to an emulated device crosses several transitions between guest, hypervisor and dom0.

For whoever runs the data centre the architectural crux is dom0: it is the domain through which the control panel, the real device drivers and the guests’ I/O backend all pass. A saturated or compromised dom0 degrades or halts every domU it hosts. Consolidation moves the risk onto a single point, it does not remove it.

The critical point: live migration

The feature that draws most attention in a seminar is live migration: moving a running domain from one physical machine to another without shutting it down. It is also the one around which the most invented figures circulate, so it pays to anchor to the source.

The mechanism is described in Live Migration of Virtual Machines (Clark and others, NSDI 2005). It proceeds by iterative copy of memory: the hypervisor transfers all the domain’s pages to the destination while the domain keeps running, then repeats the transfer for only the pages dirtied in the meantime, and iterates until the set of still-dirty pages is small enough. At that point it suspends the domain for a moment, copies the last remainder, and resumes it on the destination. The figures in the paper give service downtimes on the order of tens of milliseconds — about 60 ms in the best documented case.

Three conditions make that figure attainable, three constraints bound it. You need: shared storage between source and destination, because the disk is not copied; a high-bandwidth, low-latency network between the two nodes; compatible CPUs, because processor state is copied across. The constraints: migration does not protect against a sudden failure of the source hardware — it is a planned procedure, not a high-availability mechanism; a domain that dirties memory faster than the network can transfer it lengthens the copy phase or never converges; and the downtime measured in a lab is not what you get on a real application, with open user sessions and network timeouts.

Implications for running cost

Consolidation cuts cost items that can be measured before and after: number of physical servers, electrical draw, heat to dissipate, rack slots, hardware maintenance contracts. Those numbers are real and countable. They must sit alongside the costs that consolidation introduces: a physical machine hosting ten domains is a point of failure whose impact counts for ten services, so it demands redundancy, monitoring and recovery procedures more serious than the ones that sufficed for ten independent boxes.

For a public body the heaviest item concerns whoever maintains the system. Xen is free software, under the GPL: no licence cost for the hypervisor. But the competence to design shared storage, size dom0, choose between paravirtualisation and HVM case by case, and carry a migration through without losing data is neither free nor downloadable from a website. It is the real centre of gravity of a training initiative, and the reason an afternoon lab, where a real migration is touched by hand, is worth more than a morning of slides. The T-OSSLab seminar cycle that frames these notes, organised with the University of Pisa and the Navacchio Technology Park, is described in an insight published by noze: https://www.noze.it/en/insights/tosslab-seminars-start-2009/.

Limits

The above describes Xen on the kernels and versions available now, in early 2009; overhead percentages vary with load, type of I/O and hardware, and should not be quoted as constants. Paravirtualisation on Linux is mature; HVM is the route for unmodifiable guests, but it pays on I/O until paravirtualised drivers are added inside the guest. And none of these techniques replaces a continuity plan: virtualisation makes moving and copying a system easier, on its own it does not guarantee the system stays up.

Cover image: Block diagram of the Xen architecture: a green “XEN dom0” box with a management app and device driver at top, beside two “XEN domU”… — diagram by Radoslaw Korzeniewski, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:XEN-schema.png