By Dave Scott - 2012-09-12
[ Due to continuing development, some of the details in this blog post are now out-of-date. It is archived here. ]
On all hosts running Xen, there is a critical service called xenstore. Xenstore is used to allow untrusted user VMs to communicate with trusted system VMs, so that
If the xenstore service fails then at best the host cannot be controlled (i.e. no VM start or shutdown) and at worst VM isolation is compromised since an untrusted VM will be able to gain unauthorised access to disks or networks. This blog post examines how to disaggregate xenstore from the monolithic domain 0, and run it as an independent stub domain.
Recently in the Xen community, Daniel De Graaf and Alex Zeffertt have added support for xenstore stub domains where the xenstore service is run directly as an OS kernel in its own isolated VM. In the world of Xen, a running VM is a "domain" and a "stub" implies a single-purpose OS image rather than a general-purpose machine. Previously if something bad happened in "domain 0" (the privileged general-purpose OS where xenstore traditionally runs) such as an out-of-memory event or a performance problem, then the critical xenstore process might become unusable or fail altogether. Instead if xenstore is run as a "stub domain" then it is immune to such problems in domain 0. In fact, it will even allow us to reboot domain 0 in future (along with all other privileged domains) without incurring any VM downtime during the reset!
The new code in xen-unstable.hg lays the necessary groundwork (Xen and domain 0 kernel changes) and ports the original C xenstored to run as a stub domain.
Meanwhile, thanks to Vincent Hanquez and Thomas Gazagnaire, we also have an OCaml implementation of xenstore which, as well as the offering memory-safety, also supports a high-performance transaction engine, necessary for surviving a stressful "VM bootstorm" event on a large server in the cloud. Vincent and Thomas' code is Linux/POSIX only.
Ideally we would have the best of both worlds:
We can now do both, using Mirage! If you're saying, "that sounds great! How do I do that?" then read on...
Step 1: remove dependency on POSIX/Linux
If you read through the existing OCaml xenstored code, it becomes obvious that the main uses of POSIX APIs are for communication with clients, both Unix sockets and for a special Xen inter-domain shared memory interface. It was a fairly painless process to extract the required socket-like IO signature and turn the bulk of the server into a functor. The IO signature ended up looking approximately like:
type t
val read: t -> string -> int -> int -> int Lwt.t
val write: t -> string -> int -> int -> unit Lwt.t
val destroy: t -> unit Lwt.t
For now the dependency on Lwt is explicit but in future I'll probably make it more abstract so we can use Core Async too.
Step 2: add a Mirage Xen IO implementation
In a stub-domain all communication with other domains is via shared memory pages and "event channels". Mirage already contains extensive support for using these primitives, and uses them to create fast network and block virtual device drivers. To extend the code to cover the Xenstore stub domain case, only a few tweaks were needed to add the "server" side of a xenstore ring communication, in addition to the "client" side which was already present.
In Xen, domains share memory by a system of explicit "grants", where a client (called "frontend") tells the hypervisor to allow a server (called "backend") access to specific memory pages. Mirage already had code to create such grants, all that was missing was a few simple functions to receive grants from other domains.
These changes are all in the current mirage-platform tree.
Step 3: add a Mirage Xen "main" module and Makefile
The Mirage "main" module necessary for a stub domain looks pretty similar to the normal Unix userspace case except that it:
The Makefile looks like a regular Makefile, invoking ocamlbuild. The whole lot is built with OASIS with a small extension added by Anil to set a few options required for building Xen kernels rather than regular binaries.
... and it all works!
The code is in two separate repositories:
add-xen
branch from this OASIS fork.Example build instructions
If you want to try building it yourself, try the following on a modern 64-bit OS. I've tested these instructions on a fresh install of Debian Wheezy.
First install OCaml and the usual build tools:
apt-get install ocaml build-essential git curl rsync
Then install the OCamlPro opam
package manager to simplify the installation of extra packages
git clone https://github.com/OCamlPro/opam.git
cd opam
make
make install
cd ..
Initialise OPAM with the default packages:
opam --yes init
eval `opam config -env`
Add the "mirage" development package source (this step will not be needed once the package definitions are upstreamed)
opam remote -add dev https://github.com/mirage/opam-repo-dev
Switch to the special "mirage" version of the OCaml compiler
opam --yes switch -install 3.12.1+mirage-xen
opam --yes switch 3.12.1+mirage-xen
eval `opam config -env`
Install the generic Xenstore protocol libraries
opam --yes install xenstore
Install the Mirage development libraries
opam --yes install mirage
If this fails with "+ runtime/dietlibc/lib/atof.c:1: sorry, unimplemented: 64-bit mode not compiled in" it means you need a 64-bit build environment. Next, clone the xen stubdom tree
git clone https://github.com/djs55/ocaml-xenstore-xen
Build the Xen stubdom
cd ocaml-xenstore-xen
make
The binary now lives in xen/_build/src/server_xen.xen
Deploying on a Xen system
Running a stub Xenstored is a little tricky because it depends on the latest and greatest Xen and Linux PVops kernel. In the future it'll become much easier (and probably the default) but for now you need the following:
init-xenstore-domain
from xen-4.2/tools/xenstore
.To turn the stub xenstored on, you need to edit whichever init.d
script is currently starting xenstore and modify it to call
init-xenstore-domain /path/to/server_xen.xen 256 flask_label