Irmin 0.12 with portable filesystem watching released

Development of the Irmin Git-like data store continues (see here for an introduction). We are releasing Irmin 0.12.0 which brings support for native file-system watchers to greatly improve the performance of watches on the datastore.

Previously, an Irmin application that wanted to use watches would setup file-system scanning/polling by doing:

  let () = Irmin_unix.install_dir_polling_listener 1.0

which would scan the .git/refs directory every second. This worked in practice but was unpredictably latent (if unlucky you might wait for a full second for the watch callbacks to trigger), and disk/CPU intensive as we were scanning the full storage directory every second to detect file changes. In the cases where the store had 1000s of tags, this could easily saturate the CPU. And in case you were wondering, there are increasing number of applications (such as DataKit) that do create thousands of tags regularly, and Canopy that need low latency for interactive development.

In the new 0.12.0 release, you need to use:

   let () = Irmin_unix.set_listen_dir_hook ()

and the Irmin storage will do "the right thing". If you are on Linux, and have the inotify OPAM package installed, it will use libinotify to get notified by the kernel on every change and re-scan the whole directory. On OSX, if you have the osx-fsevents OPAM package installed, it will do the same thing using the OSX FSEvents.framework. The portable compatibility layer between inotify and fsevents comes via the new irmin-watcher package that has been released recently as well. This may also come in useful for other tools that require portable OCaml access to filesystem hierarchies.

If you are using Irmin, please do let us know how you are getting on via the mailing list and report any bugs on the issue tracker.