Introducing Functoria

For the last few months, I've been working with Thomas on improving the mirage tool and I'm happy to present Functoria, a library to create arbitrary MirageOS-like DSLs. Functoria is independent from mirage and will replace the core engine, which was somewhat bolted on to the tool until now.

This introduces a few breaking changes so please consult the breaking changes page to see what is different and how to fix things if needed. The good news is that it will be much more simple to use, much more flexible, and will even produce pretty pictures!

Configuration

For people unfamiliar with MirageOS, the mirage tool handles configuration of mirage unikernels by reading an OCaml file describing the various pieces and dependencies of the project. Based on this configuration it will use opam to install the dependencies, handle various configuration tasks and emit a build script.

A very simple configuration file looks like this:

open Mirage
let main = foreign "Unikernel.Main" (console @-> job)
let () = register "console" [main $ default_console]

It declares a new functor, Unikernel.Main, which take a console as an argument and instantiates it on the default_console. For more details about unikernel configuration, please read the hello-world tutorial.

Keys

A much demanded feature has been the ability to define so-called bootvars. Bootvars are variables whose value is set either at configure time or at startup time.

A good example of a bootvar would be the IP address of the HTTP stack. For example, you may wish to:

Set a good default directly in the config.ml
Provide a value at configure time, if you are already aware of deployment conditions.
Provide a value at startup time, for last minute changes.

All of this is now possible using keys. A key is composed of:

name — The name of the value in the program.
description — How it should be displayed/serialized.
stage — Is the key available only at runtime, at configure time, or both?
documentation — This is not optional, so you have to write it.

Imagine we are building a multilingual unikernel and we want to pass the default language as a parameter. The language parameter is an optional string, so we use the opt and string combinators. We want to be able to define it both at configure and run time, so we use the stage `Both. This gives us the following code:

let lang_key =
  let doc = Key.Arg.info
      ~doc:"The default language for the unikernel." [ "l" ; "lang" ]
  in
  Key.(create "language" Arg.(opt ~stage:`Both string "en" doc))

Here, we defined both a long option --lang, and a short one -l, (the format is similar to the one used by Cmdliner). In the unikernel, the value is retrieved with Key_gen.language ().

The option is also documented in the --help option for both mirage configure (at configure time) and ./my_unikernel (at startup time).

       -l VAL, --lang=VAL (absent=en)
           The default language for the unikernel.

A simple example of a unikernel with a key is available in mirage-skeleton in the hello directory.

Switching implementation

We can do much more with keys, for example we can use them to switch devices at configure time. To illustrate, let us take the example of dynamic storage, where we want to choose between a block device and a crunch device with a command line option. In order to do that, we must first define a boolean key:

let fat_key =
  let doc = Key.Arg.info
      ~doc:"Use a fat device if true, crunch otherwise." [ "fat" ]
  in
  Key.(create "fat" Arg.(opt ~stage:`Configure bool false doc))

We can use the if_impl combinator to choose between two devices depending on the value of the key.

let dynamic_storage =
  if_impl (Key.value fat_key)
    (kv_ro_of_fs my_fat_device)
    (my_crunch_device)

We can now use this device as a normal storage device of type kv_ro impl! The key is also documented in mirage configure --help:

       --fat=VAL (absent=false)
           Use a fat device if true, crunch otherwise.

It is also possible to compute on keys before giving them to if_impl, combining multiple keys in order to compute a value, and so on. For more details, see the API and the various examples available in mirage and mirage-skeleton.

Switching keys opens various possibilities, for example a generic_stack combinator is now implemented in mirage that will switch between socket stack, direct stack with DHCP, and direct stack with static IP, depending on command line arguments.

Drawing unikernels

All these keys and dynamic implementations make for complicated unikernels. In order to clarify what is going on and help to configure our unikernels, we have a new command: describe.

Let us consider the console example in mirage-skeleton:

open Mirage

let main = foreign "Unikernel.Main" (console @-> job)
let () = register "console" [main $ default_console]

This is fairly straightforward: we define a Unikernel.Main functor using a console and we instantiate it with the default console. If we execute mirage describe --dot in this directory, we will get the following output.

As you can see, there are already quite a few things going on! Rectangles are the various devices and you'll notice that the default_console is actually two consoles: the one on Unix and the one on Xen. We use the if_impl construction — represented as a circular node — to choose between the two during configuration.

The key device handles the runtime key handling. It relies on an argv device, which is similar to console. Those devices are present in all unikernels.

The mirage device is the device that brings all the jobs together (and on the hypervisor binds them).

Data dependencies

You may have noticed dashed lines in the previous diagram, in particular from mirage to Unikernel.Main. Those lines are data dependencies. For example, the bootvar device has a dependency on the argv device. It means that argv is configured and run first, returns some data — an array of string — then bootvar is configured and run.

If your unikernel has a data dependency — say, initializing the entropy — you can use the ~deps argument on Mirage.foreign. The start function of the unikernel will receive one extra argument for each dependency.

As an example, let us look at the app_info device. This device makes the configuration information available at runtime. We can declare a dependency on it:

let main =
  foreign "Unikernel.Main" ~deps:[abstract app_info] (console @-> job)

The only difference with the previous unikernel is the data dependency — represented by a dashed arrow — going from Unikernel.Main to Info_gen. This means that Unikernel.Main.start will take an extra argument of type Mirage_info.t which we can, for example, print:

name: console
libraries: [functoria.runtime; lwt.syntax; mirage-console.unix;
            mirage-types.lwt; mirage.runtime; sexplib]
packages: [functoria.0.1; lwt.2.5.0; mirage-console.2.1.3; mirage-unix.2.3.1;
           sexplib.113.00.00]

The complete example is available in mirage-skeleton in the app_info directory.

Since we have a way to draw unikernels, we can now observe the sharing between various pieces. For example, the direct stack with static IP yields this diagram:

You can see that all the sub-parts of the stack have been properly shared. To be merged, two devices must have the same name, keys, dependencies and functor arguments. To force non-sharing of two devices, it is enough to give them different names.

This sharing also works up to switching keys. The generic stack gives us this diagram:

If you look closely, you'll notice that there are actually three stacks in the last example: the socket stack, the direct stack with DHCP, and the direct stack with IP. All controlled by switching keys.

All your functors are belong to us

There is more to be said about the new capabilities offered by functoria, in particular on how to define new devices. You can discover them by looking at the mirage implementation.

However, to wrap up this blog post, I offer you a visualization of the MirageOS website itself (brace yourself). Enjoy!

Thanks to Mort, Mindy, Amir and Jeremy for their comments on earlier drafts.