Skip to content

Home

Jump to bottom
Joachim Schiele edited this page Jul 26, 2024 · 45 revisions

Motivation

libnix: Make cargo use nix internally as a backend using the nix c interface.

since the toolchain is a fixpoint and we benefit if that is small (having only the bare minimum needed to compile/link the crates/program) we need to have a build-environment.nix in the project's project dir. This build-environment.nix is similar to a callPackage file, a nix function which is passed in a rust toolchain and basically pkgs so it can use nix to access the compiler and c-libraries.

A central flake.nix can use overrides to change the rust toolchain for all crates in this project. Each crate dependency can also have a build-environment.nix for individual dependencies.

So every crate could have a build-environment.nix like:

  brotli_decompressor_1_3_1_ = { dependencies?[], buildDependencies?[], features?[] }: buildRustCrate {
    crateName = "brotli-decompressor";
    version = "1.3.1";
    authors = [ "Daniel Reiter Horn <[email protected]>" "The Brotli Authors" ];
    sha256 = "022g69q1xzwdj0130qm3fa4qwpn4q1jx3lc8yz0v0v201p7bm8fb";
    crateBin = [ {  name = "brotli-decompressor"; } ];
    inherit dependencies buildDependencies features;
  };

which calls pkgs/build-support/rust/build-rust-crate/default.nix:

# Code for buildRustCrate, a Nix function that builds Rust code, just
# like Cargo, but using Nix instead.
#
# This can be useful for deploying packages with NixOps, and to share
# binary dependencies between projects.

{ lib, stdenv, defaultCrateOverrides, fetchCrate, rustc, rust, cargo, jq }:

let
    # This doesn't appear to be officially documented anywhere yet.
    # See https://github.com/rust-lang-nursery/rust-forge/issues/101.
    target_os = if stdenv.hostPlatform.isDarwin
      then "macos"
      else stdenv.hostPlatform.parsed.kernel.name;
...

A flake.nix can:

When you enter the directory /home/joachim/myrustproject you will see the rust tools like: cargo-clippy / cargo-fmt / cargo / rust-analyzer / rust-gdb / rust-lldb / rustc / rustdoc / rustfmt. When you leave this directory the tools won't be in the shell anymore.

In your top level project you have a flake.nix:

... FIXME override rust with overlay so we can pick rust version per project

This myrustproject specific .envrc should only contain the bare minimum to describe the environment needed for the rust project. So a flake could make use to build an output from the rust project but if said flake could contain several rust projects and each would have a specific definition of rust compiler settings.

The same set of tools will be used by cargo when spawning a build, this gives use reproducability in builds since this might require more than just cargo/rustc. For instance a c compiler or some other useful third-party tool.

See also:

Example usages of cargo/crates

Rust only crate:

Rust+C: Bundling c code in the create:

Rust+C: External dependencies from the system:

Build creates using nix abstractions

Working branch

https://github.com/nixcloud/cargo/tree/libnix-0.79.0

parts of both folders ~/.cargo and target have been replaced by the nix store.

Let's see what is in them!

~/.cargo

[nixos@nixos:~/cargo]$ du -sh ~/.cargo
239M    /home/nixos/.cargo

[nixos@nixos:~/cargo]$ ls ~/.cargo/
registry

[nixos@nixos:~/cargo]$ ls ~/.cargo/registry/
cache  CACHEDIR.TAG  index  src

cargo/target

The example is the cargo project itself, the tool coming with the rust toolchain, since we hack on it.

[nixos@nixos:~/cargo]$ du -sh target/
2.3G    target/

[nixos@nixos:~/cargo]$ ls target/
CACHEDIR.TAG  debug

[nixos@nixos:~/cargo]$ ls target/debug/
build  cargo  cargo.d  deps  examples  incremental  libcargo.d  libcargo.rlib

Plan

libnix backend in cargo needs to get these things working:

  • no use of ~/.cargo, everything in /nix/store
  • target/debug/* needs to come from /nix/store, eventually symlinked or transparently from the store

figure out what these are:

  • target/debug/build

.fingerprint

target/debug/.fingerprint -> https://doc.rust-lang.org/nightly/nightly-rustc/cargo/core/compiler/fingerprint/index.html

[nixos@nixos:~/cargo]$ ls target/debug/.fingerprint/der-d0714c795318982b/
dep-lib-der  invoked.timestamp  lib-der  lib-der.json

[nixos@nixos:~/cargo]$ cat target/debug/.fingerprint/der-d0714c795318982b/lib-der
6b2cf10d51d11828

[nixos@nixos:~/cargo]$ cat target/debug/.fingerprint/der-d0714c795318982b/lib-der.json  | jq
{
  "rustc": 18217185010275080438,
  "features": "[\"alloc\", \"oid\", \"pem\", \"std\", \"zeroize\"]",
  "declared_features": "",
  "target": 4346314333490370059,
  "profile": 12206360443249279867,
  "path": 16879011525611689376,
  "deps": [
    [
      1724196337906130016,
      "pem_rfc7468",
      false,
      5466269655917579261
    ],
    [
      2676387858222833023,
      "const_oid",
      false,
      2852573652909154018
    ],
    [
      15098350142499636151,
      "zeroize",
      false,
      14469829910961900562
    ]
  ],
  "local": [
    {
      "CheckDepInfo": {
        "dep_info": "debug/.fingerprint/der-d0714c795318982b/dep-lib-der"
      }
    }
  ],
  "rustflags": [],
  "metadata": 12456048820742377390,
  "config": 2202906307356721367,
  "compile_kind": 0
}

[nixos@nixos:~/cargo]$ cat target/debug/.fingerprint/der-d0714c795318982b/dep-lib-der

finally:

  • get a crate compiled without dependencies
  • get one with nested dependencies
  • get one with c library usage (like cargo itself)

BuildContext

Some fields that might be interesting:

let bcx : BuildContext = create_bcx(ws, options, &interner)? // Line 153 in cargo/ops/cargo_compile/mod.rs
bcx.ws.gctx.home_path;
bcx.ws.gctx.cwd;
bcx.ws.gctx.ws_roots;
bcx.ws.current_manifest;
bcx.ws.packages.gctx;
bcx.ws.packages.packages; // HashMap<PackageId, LazyCell<Package>>

Compilation outcomes

Where do we need to grab the compilation results that we want to move into the nix store?

https://docs.rs/cargo/latest/src/cargo/core/compiler/build_runner/mod.rs.html#135

Can we just use the Compilation outcome?

With Rust nightly we can see build plan:

cargo build -Z unstable-options --build-plan | jq
Clone this wiki locally