buildRustCrate is a tool I wrote in the Nix programming language to share build products across crates and across versions of a single crate when compiling Rust code. The initial motivation was to speed up deployment times when working on large projects with lots of dependencies, such as the Pijul Nest.

What are Nix and NixOS?

NixOS is a purely functional Linux distribution aimed at guaranteeing reproducible builds. Here, “purely functional” means declarative, in the sense that the user writes code that gets compiled into the appropriate configuration files and installed packages.

This idea works extremely well, even though it does have a few caveats, mostly related to security: for instance, passwords and SSL keys are not stored in the Nix programs people usually write, for security reasons. This can of course be changed when required, just by writing some Nix code.

Packaging stuff for Nix

Ultimately, a “package” is called a derivation in Nix terms. I like to see derivations as just records with a bunch of fields that need to be set (such as the derivation’s dependencies), and some others that are computed automatically (such as the derivation’s output directory). Once compiled (via a tool called nix-build), derivations have two parts:

• One file stored in a path of the form /nix/store/5k7aknlnnrxz6dzav992wljy7aw8gqzv-carnix.drv, containing all the information required to build a package, including precise versions, commits, dependencies, etc.

• A directory stored in a path of the form /nix/store/4zc3rsnsr3lkkfmkam9blkw5h8j4hvxg-rust_carnix-0.5.0, containing the actual files.

Also, derivations can reference other derivations: for instance, the upstream tarball itself can be a derivation.

An important point is that for any two packages A and B, A is either totally independent from B, or else A references B, or B references A (in the files ending in .drv).

This is achieved by a set of strict rules enforced by nix-build to avoid any side effects when building a package, and yields a system that never “ages”: installing new packages, reinstalling the whole system, or upgrading (some packages or the whole system), are all effectively the same operation. Configuration files are all in /nix/store/…, generated by nix-build, and old versions are garbage-collected when no longer needed.

Also, this is a pretty nice framework DevOps, since configuration can be transferred across machines without the need for virtual machines, sandboxes and complex packaging systems. The Nix community has actually produced a tool called NixOps to deploy machines in the cloud using the same principles.

buildRustCrate

However, when insisting on referential transparency (i.e. when avoiding side-effects), Nix can be sometimes a little frustrating. For instance, when I started deploying the Nest using NixOps, nix-build kept recompiling all dependencies every time, even when nothing had changed. This was due to the fact that Nix was unable to make any assumption on how Cargo worked, and even on the fact that Cargo was a package manager.

After too much time spent waiting for my machine to finish compiling, I decided to write my own build system in Nix. Several months later, with many iterations and great feedback from the NixOS team, we got a pretty solid solution, using two tools, Carnix and buildRustCrate.

Carnix

Carnix is a small tool written in Rust and available both on crates.io and in nixpkgs (nix-env -iA nixos.carnix, once the NixOS continuous integration system is done compiling it).

Carnix can be used to compile the Cargo.lock of a crate (as generated by Cargo) into a Nix file, ready to be processed by nix-build. It knows how to deal with dependencies, sources, and features. The generated output looks like:

with import <nixpkgs> {};
let release = true;
    verbose = true;
    hello_0_1_0_ = { dependencies?[], buildDependencies?[], features?[] }: buildRustCrate {
      crateName = "hello";
      version = "0.1.0";
      src = ./.;
      inherit dependencies buildDependencies features release verbose;
    };
    libc_0_2_33_ = { dependencies?[], buildDependencies?[], features?[] }: buildRustCrate {
      crateName = "libc";
      version = "0.2.33";
      sha256 = "1l7synziccnvarsq2kk22vps720ih6chmn016bhr2bq54hblbnl1";
      inherit dependencies buildDependencies features release verbose;
    };

in
rec {
  hello_0_1_0 = hello_0_1_0_ {
    dependencies = [ libc_0_2_33 ];
  };
  libc_0_2_33 = libc_0_2_33_ {
    features = [ "use_std" ];
  };
}

A nice feature of this is that crates are now defined in the same language as any other library available on NixOS, which means that we can add external dependencies automatically, making the build process for crates that depend on native libraries (such as OpenSSL) even easier to use than with Cargo only.

buildRustCrate

The generated output calls buildRustCrate, which is a small file written in a mixture of Bash and Nix, that tries to mimic as faithfully as possible (according to our tests) what cargo does.

See this file on GitHub.