TL;DR: New config-as-code immutable build system in Rust and OCaml, see https://nest.pijul.com/pmeunier/elpe
Elpe is a library to describe build environments from other Linux distributions, and run scripts in these environments in containers. This project tries to bridge the gap between a purely functional, almost fully bootstrapped Linux distributions like Nix, and a more mainstream package manager like Debian and Ubuntu.
I started thinking about this topic when trying to add a CI system to the open source Nest, which I’ll blog about in a near future.
There is a trade-off between two widely different visions of what software distribution should look like on Linux:
On one end of the spectrum, the Debian manifesto states that compiling packages is an expert task which shouldn’t have to be done by the end users of a Linux system.
The tools created to implement this vision (dpkg, apt, debconf…) go a long way, but I feel they’ve always fell short of freeing users from the tedious manual work required to update their system: for example, if you hack your configuration for long enough or with enough concurrent hands, you will often end up having to reinstall from scratch after a while.
On the other end of the spectrum, the NixOS project wants to create a fully reproducible system, described almost entirely as code in a purely functional programming language.
In this vision, compilation is no longer an expert task since it is specified to a large extent by code in the Nix language: since builds are isolated from the context, anyone can perform them, sometimes even without realising that they are. The distinction between source and binary packages is no longer relevant.
Add atomic updates of the entire configuration and a global cache of build products to the game, and you get the really cozy “safety net for system administration” that NixOS is.
Many other projects, such as Fedora Silverblue, Docker or Gentoo solve this trade-off in various ways, making different assumptions I won’t dive into.
On a personal note, I’ve been a Debian user between 2000 and 2014, and a NixOS user and contributor since then (and never reinstalled my laptop ever since then, and it’s still my daily driver).
The two solutions I am going to talk about now are more than 20 years old. Debian started in 1993, and NixOS in 2003.
One massive problem with Debian/Ubuntu is the lack of build reproducibility and atomic updates. This means that, in particular:
This implies that getting a full view of what is on your computer takes a lot of rigorous work (a bit like when working with Git, but at an even larger scale).
NixOS solves these quite elegantly, but faces multiple subtle, mostly non-technical problems:
Supply chain security: Nixpkgs seems at first glance like an excellent solution to supply chain issues, since you get a full explicit view, described in the Nix language, of how all the software installed on your system was built. However, the practice is quite different, due to the massive size of the existing code and the speed at which things are proposed and get merged every day. The same thing that made Nix so easy to contribute to and hence so successful also makes Nixpkgs security a moving target.
Governance: everybody can start contributing packages in a matter of minutes (easiest experience I’ve ever had when contributing to a project I care about!). Also, depending on who you ask, “Nix” may mean the language, the implementation, the Linux distribution or the package repository. As a consequence, many people have different interests and opinions about how processes should work and who should be in charge. Should you be allowed to self-approve your changes to Nixpkgs, in particular when introducing controversial features? To make changes in the core design of Nix itself without asking anyone? To decide how contributors want to interact with corporate actors, on their behalf?
I am in no way implying that these questions have easy answers for everybody (I would answer no to all, and have answered no for all my other open source projects, but I know this is just my opinion).
Software security: beyond processes and supply chain considerations, the technical part of software security require extremely broad knowledge, and finding the resources to look after a codebase of the size of Nixpkgs is hard.
The Nix language is not statically typed and not compiled, making it hard to learn and scale to a larger organisation. In my opinion, there are three stages of learning Nix, which can be identified by the errors you face the most:
At all stages, you may also experience regular compilation errors, which is the only errors we should actually care about when writing Nix. Static typing would get rid of beginner errors, while a simpler language could make overengineering less appealing.
Finally, there is one element of design I don’t like much, which is the many “brain teasers” required to make the Nix language do more clever things, such as fixpoints everywhere. While these are cool in other contexts (I am definitely a big Haskell fan), I feel they tend to distract otherwise talented engineers into showing off how smart they are, which may in turn rub their colleagues the wrong way and make them think Nix is just a fad.
I feel that a language simple enough to make these behaviours unappealing would solve many adoption problems.
When thinking about the CI system for the Nest, I wanted to explore a different take on the trade-off I explained above. I don’t really think the problems of Nix and Nixpkgs stated above come from bad design or engineering, so I’m definitely not planning on doing “Nix, but right” (besides, other projects are already doing that, see Tvix and Lix).
The goal of Elpe is to use the excellent idea (taken from Nix) of hard-coding paths into bash scripts and running them in containers, but using Ubuntu packages instead of bootstrapping the entire package collection, making the separation between my own build code and the system’s clearer: we don’t even use the same parts of the Elpe API to describe these two operations.
Also, I wanted to use that as an opportunity to rethink some of the bits I don’t quite like, such as the suboptimal language design and some security issues.
I’ve explained some of the design choices in the project’s README.
This is a Linux project for now, but could easily be extended with other platforms.
Elpe could probably be ported to other operating systems, with some trade-offs in terms of performance and security due to the lack of process namespacing. Please reach out if you’re interested in this.
Here’s a “Hello, world!” package compiled using Elpe.
First write the following Makefile:
all:
gcc -o hello hello.c
install:
install -D --mode 555 hello ${DESTDIR}/usr/bin/hello
And a file named hello.c:
#include <stdio.h>
int main() {
printf("Hello, world!\n");
}
Then, write the following OCaml file, named elpe.ml:
open Elpe
open Lwt.Syntax
let _ =
build
(object (self)
inherit std_derivation
method name = "test"
method! src = self#local_src "."
method! build_inputs =
Lwt.return
[
(ubuntu "gcc" :> derivation);
(ubuntu "libc6-dev" :> derivation);
(ubuntu "make" :> derivation);
(ubuntu "coreutils" :> derivation);
(ubuntu "libstdc++-13-dev" :> derivation);
]
end)
Then, start the Elpe daemon:
$ sudo elpe-server -c config.toml
With the appropriate config.toml (example in the main Elpe
repo). You can leave the daemon running.
Finally, build the package:
$ elpe build
Feel free to reach out in the Pijul Zulip if you’re interested in helping. I’ve listed a few of the items that need help in the project’s README.