The MirageOS Blogon building functional operating systems
MirageOS Spring 2019 hack retreat roundup
MirageOS Spring Hack Retreat, Marrakesh 2019
Early March 2019, 31 MirageOS hackers gathered again in Marrakesh for our bi-annual hack retreat. We’d like to thank our amazing hosts, and everyone who participated on-site or remotely, and especially those who wrote up their experiences.
On this retreat, we ate our own dogfood, and used our MirageOS DHCP, recursive DNS resolver, and CalDAV unikernels as isolated virtual machines running on a PC Engines APU with FreeBSD as host system. The CalDAV server persisted its data in a git repository on the host system, using the raw git protocol for communication, the smart HTTP protocol could have been used as well. Lynxis wrote a detailed blog post about our uplink situation.
Lots of interesting discussions took place, code was developed, knowledge was exchanged, and issues were solved while we enjoyed the sun and the Moroccan food. The following list is not exhaustive, but gives an overview what was pushed forward.
Imagelib
Imagelib is a library that can parse several image formats, and eye-of-mirage uses it to display those images in a framebuffer.
During the retreat, imagelib was extended with support for the BMP format, it’s build system was revised to split off Unix-dependent functionality, and preliminary support for the GIF format was implemented.
ActivityPub
ActivityPub is an open, decentralized social networking protocol, as used by mastodon. It provides a client/server API for creating, updating, and deleting content, and a federated server-to-server API for notifications and content delivery. During the retreat, an initial prototype of a protocol implementation was drafted.
opam
Opam, the OCaml package manager, was extended in several directions:
- External (OS package system) dependency integration
- Interleaving download with build/install actions
- Generalisation of the job scheduler
- JSON serialisation, including crowbar round-trip tests
- Plugin evaluating (binary) reproducibility of opam packages
- some smaller cleanup PRs (return values, locking code)
marracheck
Work was started on a new utility to install as many opam packages as possible on a machine (there just wasn’t enough choice with opam-builder, opamcheck and opam-check-all). It uses opam-lib and Z3 to accomplish this.
Conex
Conex is used for signing community repositories, esp. the opam-repository. Any opam package author can cryptographically sign their package releases, and users can verify that the downloaded tarball and build instructions are identical to what the author intended.
Conex has been developed since 2015, but is not yet widely deployed. We extended opam-publish to invoke the conex_targets
utility and sign before opening a pull request on the opam-repository.
SMTP
The simple mail transfer protocol is an Internet standard for sending and receiving eMail. Our OCaml implementation has been improved, and it is possible to send eMails from OCaml code now.
HTTP2
The hypertext transfer protocol is an Internet standard widely used for browsing the world wide web. HTTP 1.1 is a line-based protocol which was specified 20 years ago. HTTP2 is an attempt to fix various shortcomings, and uses a binary protocol with multiplexing, priorities, etc. An OCaml implementation of HTTP2 has been actively worked on in Marrakesh.
Irmin
Irmin is a distributed database that follows the same design principles as git. Soon, Irmin 2.0 will be released, which includes GraphQL, HTTP, chunk support, and can use the git protocol for interoperability. Irmin provides a key-value interface for MirageOS.
OCaml compiler
Some hints on type errors for int literals and int operators were developed and merged to the OCaml compiler.
# 1.5 +. 2;; ^ Error: This expression has type int but an expression was expected of type float Hint: Did you mean `2.'? # 1.5 + 2.;; ^^^ ^ Error: This expression has type float but an expression was expected of type int Line 1, characters 4-5: Hint: Did you mean to use `+.'?
Also, the whole program dead code elimination PR was rebased onto trunk.
BGP / lazy trie
The mrt-format library which can parse multi-threaded routing toolkit traces, has been adapted to the modern OCaml ecosystem. The border gateway protocol (BGP) library was slightly updated, one of its dependencies, lazy-trie was adapted to the modern ecosystem as well.
Xen PVH
Xen provides several modes for virtualization. MirageOS’s first non-Unix target was the para-virtualized (PV) mode for Xen, which does not require hardware support from the hypervisor’s host operating system but has a weak security profile (static mapping of addresses, large attack surface). However, PV mode provides an attractive target for unikernels because it provides a simple software-based abstraction for dealing with drivers and a simple memory model; this is in contrast to hardware-virtualization mode, which provides greater security but requires more work from the guest OS.
A more modern virtualization mode combining the virtues of both approaches is PVH (formerly referred to as HVMLite), which is not yet supported by MirageOS. Marek Marczykowski-Górecki from the QubesOS project visited to help us bring PVH support to the unikraft project, a common platform for building unikernels which we hope to use for MirageOS’s Xen support in the future.
During the retreat, lots of bugs porting MirageOS to PVH were solved. It boots and crashes now!
Learn OCaml as a unikernel
The platform learn OCaml embeds an editor, top-level, and exercises into a HTTP server, and allows students to learn OCaml, and submit solutions via the web interface, where an automated grader runs unit tests etc. to evaluate the submitted solutions. Teachers can assign mandatory exercises, and have an overview how the students are doing. Learn OCaml used to be executable only on a Unix host, but is now beeing ported into a MirageOS unikernel, executable as a standalone virtual machine.
Network device driver (ixy)
The ixy network driver supports Intel 82599 network interface cards, and is implemented in OCaml. Its performance has been improved, including several failing attempts which degraded its performance. Also, it has been integrated into the mirage tool and is usable as a mirage-net implementation.
DNS client API
Our proposed API is described here. Unix, Lwt, and MirageOS implementations are already available.
mirage-http unified HTTP API
Since we now have two HTTP servers, cohttp and httpaf, in OCaml and MirageOS available, the new interface mirage-http provides a unified interface, and also supports connection upgrades to websockets.
cstruct capabilities
We use cstruct, a wrapper around OCaml’s Bigarray, quite a lot in MirageOS. Until now, cstruct is a readable and writable byte array. We used phantom types to add capabilities to the interface to distinct read-only and write-only buffers.
patch
An OCaml implementation to apply unified diffs. This code has been extracted from conex (since we found some issues in it), and still needs to be fixed.
Statistical memory profiler
Since 2016, Jacques-Henri Jourdan has been working on a statistical memory profiler for OCaml (read the OCaml 2016 paper). An Emacs user interface is available since some years. We integrated statmemprof into MirageOS unikernels using the statmemprof-mirage library, marshal the data via TCP, and provide a proxy that communicates with Emacs over a Unix domain socket, and the unikernel.
P2Pcollab
P2Pcollab is a collection of composable libraries implementing protocols for P2P collaboration. So far various P2P gossip protocols has been implemented. At this retreat the focus was on a gossip-based publish-subscribe dissemination protocol. Future plans include building P2P unikernels and adding P2P pub/sub sync functionality to Irmin.