The MirageOS Documentation

Agenda

• mirage-net-xen race issue
• Quality and test infrastructure
• Deployment automation
• Irmin progress
• Steps to Mirage.io
• Pioneer Projects
• 3.0 Planning

Attendees: Amir Chaudhry (chair), Thomas Gazagnaire, Heidi Howard, David Kaloper, Thomas Leonard, Anil Madhavapeddy, Hannes Mehnert, Richard Mortier, Dave Scott, Mindy Preston, Nicolas Ojeda Bar and Magnus Skjegstad

Notes

mirage-net-xen race issue

There seems to be a dead-lock/race issue somewhere in the mirage-net-xen 1.3.0 release (mirage/mirage-net-xen#20). We originally thought this might be something to do with one of Dave's patches but it's not clear that's really the case. Currently, we're not even sure if it's definitely a v1.3 issue but things do seem to work under v1.2. It's even possible that it could be a dom0 problem. A reproduction of the issue from others would be quite helpful and Magnus mentioned he would give this a shot. Either way, this tells us we need more unit tests, as we've previously discussed.

Quality and test infrastructure

Last time, we discussed whether we may be able to do certain things with OPAM remotes (e.g like splicing between remotes). After a chat with Louis, the lead maintainer of OPAM, it seems this would not be a straightforward thing to do — which means it's not an avenue we can explore for now.

What we can realistically do in the short term is improve our own practices. We have all the pieces in place so it's a matter of using them and getting into the habit of incorporating them into our normal process. Something that would really help is a QuickCheck style library. [It seems QuickCheck is in OPAM but code maintenance is quite unclear - Amir]

It was raised that we have a nice modular network stack which is abstracted away from things underneath. There may be a way to use this approach to benefit more systematic testing. This is relevant to the trace-checking work that DavidK is working on and in addition, trying to get pcap input would be good. Many of these things are tied to the performance framework and getting to something that would let us run Iperf everyday would be great.

Anil proposed forming a breakout squad with himself, DavidK and Mindy who would think about testing and performance issues — Hannes was also suggested but he declined, stating dryly that he doesn't really care about performance (resulting in laughter from everyone else). Someone else who's done things in this area is Luke Dunstan (another contributor), so it's worth being aware of those efforts.

In general, the approach should be that we should work on the simple things before the more complicated things. However, it's not always clear what is meant by 'simple'. To help clarify this, we should constrain it to things within a single unikernel — no aspects of dom0 or anything else.

Since testing and quality is such an important issue, Amir will keep it on the agenda for each call.

Deployment automation

Amir's interested to see what solutions/scripts people have created around how we deploy unikernels. He's particularly interested in automated processes that pick up from where his previous Jekyll to Unikernel post left off (i.e. with a unikernel committed into a deployment repo).

After a quick poll of attendees, it seems that the deployment processes people use is quite diverse. Mindy has a set of scripts for EC2, Anil uses a cronjob and Dave has used XAPI. Each of these seems bespoke and it's unlikely anything is shared between them (compare with the TravisCI set up, which has become quite consistent across projects).

Amir will look over these to see where things stand and will likely write up a blog post about his ideal deployment workflow. Ultimately, his desire is to set up an end-to-end system such that a git push to a repo will finish with a newly built unikernel being started on one of our Bytemark machines.

Irmin progress

ThomasL is making a browser-based ToDo app that uses Irmin with local HTML5 storage. This is to replace his current task management system (also browser-based). At present, it's not talking to any server (so no backup), is fully in the browser and commits on every change. It's not too difficult to do sync but it is difficult to use any git tools (due to lack of support) — ThomasL is not using the Git backend yet. One very useful thing would be getting the SHA-1 code implemented elsewhere. We can either expose it from where it is currently or put the code elsewhere. The next step for ThomasG is to compile zlib library to Xen.

When this system works it would be really useful for Real World OCaml commenting system. It would allow the comment to be stored locally first and then synced to the GitHub 'back-end' — thus solving a problem some users have reported about comments being lost in transit.

The repository with ThomasL's app is cuekeeper — though be aware that it doesn't do anything yet! There's also a useful thread on the mailing list, which has both Thomas' involved, where questions about the library and API have been discussed. That thread is likely useful for anyone wanting to explore Irmin, with the caveat that it's alpha/beta stage and there have almost certainly been changes pushed during the discussions (email thread is: "Irmin API newbie questions").

Steps to Mirage.io

Amir would like to switch to using mirage.io as the primary domain (it currently redirects to openmirage.org - so he's using it already). This is more than just a straightforward record change as we'll begin to use more of our other tools as part of the underlying site infrastructure (e.g. DNS zone). To this end, Anil has been reinstalling the second Bytemark machine in order to run a version of Xenserver. We will also need to sort out certificates as we should be running HTTPS for the site.

Pioneer Projects

It seems there's been caution from new users about taking on Pioneer Projects as the meaning of the difficulty levels is a little vague. This partly relates to expected knowledge of OCaml and also domain-specific expertise (e.g networking, security, storage, etc.). Our thoughts are that trying to learn both OCaml and a new area as part of a Pioneer Project will be quite challenging. Hence the suggestion is for a newcomer to pick an area that they're familiar with so that they have some grounding and can learn how we've implemented things. If the projects listed do not overlap with someone's existing area of knowledge then that person should mention their experience on the mailing list so we can consider new categories of projects.

If someone only has limited experience from elsewhere to draw on, and is completely new to OCaml, then we should point to existing resources so that they can get up speed (e.g. Real World OCaml, etc).

We must take care with the difficulty levels as what we consider to be a 2-star project from our perspective may be perceived as quite challenging by the person undertaking the project. This could have a detrimental effect if people think everything is actually harder than advertised.

Having said this, we really do need more 1-star projects. There are likely many things we've thought about doing that we haven't quite got around to. Please look through your issue trackers in case it helps you come up with something. One suggestion was that writing a CLI client front-end for something like the IMAP server, would be useful and straightforward. Indeed, CLIs for many other libraries would be valuable, including TLS, Syndic etc.

Anil will put together a simple cohttp project, ThomasG already added a 2-star project and Amir will write an explanatory page to go alongside the current Pioneer Projects page to describe the thoughts above.

3.0 Planning

Amir's put together a Roadmap page on the wiki, where we can (1) collect thoughts about what we might like to see in future versions of MirageOS and (2) filter that list into an agreed set of efforts for the next major release. Note that these are two separate processes and we should take care not to conflate them. Essentially, one is about ideation/creativity and the other is about decisive filtering and prioritisation. Hopefully, it's self-evident as to why it can be difficult to do both in one step (especially with large groups of people).

It's worth pointing out that the roadmap is not a feature-list. It's meant to be a description of the areas where we think we should focus efforts. This lends itself to defining clearer goals which we can collectively commit to — knowing that our efforts will be in aligned with everyone else's. It may help to think of specific scenarios if the above description feels too broad.

Some thoughts from the attendees included:

• Quality/Testing is a feature
• Treat 3.0 as the deployment release — i.e. will run in 'production' for a long time so act accordingly
• Could think of this as the one that ties up all the loose ends
• Improve the docs and onboarding process
• Better integration between components ("end up back in n-squared space")
• Javascript

This agenda item will be brought up once a month and we will converge on the efforts we want to focus on for the 3.0 release. This may happen very quickly or it may take more discussion but either way, it will be recorded in these notes.

--

AoB

• Amir is mentoring the Static Site Pioneer Project with David Sheets and someone has stepped forward to work on this (yay!) — will report back as things progress

• OPAM testing: Beta 3 is now out and should be stable so please do try it out!

• The next call is scheduled for Wednesday, 11th March - Please add any agenda items you wish to discuss in advance and refer to the mailing list for actual details a day or so in advance.