dejavu-fonts-full downloads from binary cache but does not build on NixOS

[nh2]

I found at least one package that is downloadable from cache.nixos.org but fails with a linker error when built in the NixOS sandbox:

dejavu-fonts-full downloads successfully from the cache, but building one of its dependencies (fontforge) from source fails with a linker error.

To reproduce

I've reproduced it on a from-scratch AWS machine with AMI ID nixos-19.03.git.c7a57ae6b4e-x86_64-hvm-ebs (ami-04c7d80f45a20080f).

Steps:

nix-env -iA nixos.git
git clone --recursive https://github.com/nh2/static-haskell-nix.git
cd static-haskell-nix
git checkout eb48d1ebd7f3bcc36d0316c96c2f5058f12d70c3
git submodule update --init --recursive

NIX_PATH=nixpkgs=nixpkgs nix-instantiate survey/default.nix -A haskellPackages.darcs
nix-store -r /nix/store/mxkwknn3s8d240wqhib06mx991fqmq27-dejavu-fonts-full-2.37.drv

The above prints copying path '/nix/store/h379maqxw7skwgryfxlk576dkfrfwjrn-dejavu-fonts-full-2.37' from 'https://cache.nixos.org'.... Let's build it from source:

nix-store -r /nix/store/mxkwknn3s8d240wqhib06mx991fqmq27-dejavu-fonts-full-2.37.drv --check

This will fail with linker error:

copying path '/nix/store/v7wik3xar8rkap69d5smm047rfcigq4p-source' from 'https://cache.nixos.org'...
copying path '/nix/store/2fqaj11hwhmcdz7bv74yzlnc775ajph5-Font-TTF-1.06.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/zafxl6c69mz9mhbnczy2aaryd7vl1nvc-IO-String-1.08.tar.gz' from 'https://cache.nixos.org'...
building '/nix/store/j2b6ji1xxcd7mfa7qm4kn8f8b5mpxh30-fontforge-20190413.drv'...
...
  CCLD     fontforge
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: warning: ld-linux-x86-64.so.2, needed by /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0, not found (try using -rpath or -rpath-link)
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `__libc_stack_end@GLIBC_2.2.5'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_dl_make_stack_executable@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libc.so.6: undefined reference to `_dl_exception_create@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libc.so.6: undefined reference to `_dl_find_dso_for_object@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_dl_deallocate_tls@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_dl_allocate_tls_init@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_rtld_global@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_dl_get_tls_static_info@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libc.so.6: undefined reference to `_dl_argv@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `__tunable_get_val@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_rtld_global_ro@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libdl.so.2: undefined reference to `_dl_rtld_di_serinfo@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libpthread.so.0: undefined reference to `_dl_allocate_tls@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libc.so.6: undefined reference to `__libc_enable_secure@GLIBC_PRIVATE'
/nix/store/ljx5sayk6w0bpd1ahcf0rm5fj55cqafd-binutils-2.31.1/bin/ld: /nix/store/xqs95fqkjb1kd102yjv5h5q57gcsafb3-glibc-2.27/lib/libc.so.6: undefined reference to `__tls_get_addr@GLIBC_2.3'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:2114: fontforge] Error 1
make[3]: Leaving directory '/build/fontforge-20190413/fontforgeexe'
make[2]: *** [Makefile:2892: all-recursive] Error 1
make[2]: Leaving directory '/build/fontforge-20190413/fontforgeexe'
make[1]: *** [Makefile:1875: all-recursive] Error 1
make[1]: Leaving directory '/build/fontforge-20190413'
make: *** [Makefile:1712: all] Error 2
builder for '/nix/store/j2b6ji1xxcd7mfa7qm4kn8f8b5mpxh30-fontforge-20190413.drv' failed with exit code 2

(The static-haskell-nix clone isn't strictly necessary but I'm using it to easily get the .drv in question.)

What's going on?

Shouldn't all packages on cache.nixos.org also build from source on NixOS?

[nh2]

This might be related to #61422

[nh2]

OK, indeed #61422 is the solution.

Before commit e9784b1, dejavu-fonts-full was a fixed-output derivation.

That meant that even if its dependencies like fontforge changed (e.g. because that was overridden to compile against musl like I do), it wouldn't build the fonts from source, but fetch them from cache.nixos.org. Not having to build the fonts meant that fontforge didn't even have to be built at all, so we didn't notice that it doesn't compile against musl, even though it was in the dependencies (nix-store -qR of the .drv).

In conclusion: fontforge probably never built successfully against musl, it was just unobservable until e9784b1. That's why upgrading static-haskell-nix to latest nixpkgs master, which had that fix, exposed the problem.