Darwin platform versions

[thefloweringash]

Motivation for this change

Implement the reproducibility of #77632, without the warnings that were introduced by cctools (#93596 (comment)), with support for cross compilation by moving sdk versions and deployment targets out of the darwin stdenv and onto targetPlatform.

Effectively reverts: #110140 by @holymonson. When working on aarch64-darwin the SDK version provided (and embedded in LC_BUILD_VERSION) controls some kind of compatibility layer, see gist, so I think we need to support an idea of "SDK version" even though nixpkgs generally treats darwin as a set of independent and overridable packages.

Also tries to define a platform definition structure for all darwin targets including ios. Currently these have platforms have equivalent attributes which I would like to unify with the rest of darwin. The extra configuration here pkgs/os-specific/darwin/xcode/sdk-pkgs.nix should be able to entirely removed, and instead be provided by the standard darwin stdenv and binutils.

Original PR: #77632
Related: #103053, #83180
Motivation: #105026 (comment)

Tested:

• Default stdenv (clang 7)
• stdenv with clang 10
• stdenv with clang 11 (using llvmPackages_11.compiler-rt: fix build on darwin #111701)

Untested:

• ios cross

---

Motivating test case

The following set of test cases should include no information from the building host and produce no linker warnings.

See results in $(nix-build version-test-case.nix)/summary.

version-test-case.nix

let
  pkgs = import ./. {};
  inherit (pkgs) lib;

  ## Default stdenv
  stdenv = pkgs.stdenv;

  ## With clang 11
  # stdenv = pkgs.overrideCC pkgs.stdenv pkgs.buildPackages.clang_11;

  ## With clang 10
  # stdenv = pkgs.overrideCC pkgs.stdenv pkgs.buildPackages.clang_10;

  source = pkgs.writeText "main.c" ''
    int main() {}
  '';

  tests = {
    clangDefault = ''
      $CC -v -o test ${source}
    '';

    clangExplicit = ''
      $CC -v -mmacos-version-min=10.14 -o test ${source}
    '';

    ldDefault = ''
      $CC -o test.o -c ${source}
      $LD -L${pkgs.darwin.Libsystem}/lib -lSystem -o test test.o
    '';

    ldMacosVersion = ''
      $CC -o test.o -c ${source}
      $LD \
        -macos_version_min 10.14 \
        -L${pkgs.darwin.Libsystem}/lib -lSystem -o test test.o
    '';

    ldBothLegacyFlags = ''
      $CC -o test.o -c ${source}
      $LD \
        -macos_version_min 10.14 \
        -sdk_version 10.14 \
        -L${pkgs.darwin.Libsystem}/lib -lSystem -o test test.o
    '';

    ldSdkVersion = ''
      $CC -o test.o -c ${source}
      $LD \
        -sdk_version 10.14 \
        -L${pkgs.darwin.Libsystem}/lib -lSystem -o test test.o
    '';

    ldPlatformVersion = ''
      $CC -o test.o -c ${source}
      $LD \
        -platform_version macos 10.14 10.14 \
        -L${pkgs.darwin.Libsystem}/lib -lSystem -o test test.o
    '';
  };

  buildTest = name: testScript: stdenv.mkDerivation {
    inherit name testScript;
    passAsFile = [ "testScript" "buildCommand" ];

    buildCommand = ''
      set -u
      echo $testScriptPath
      mkdir $out
      NIX_DEBUG=3 $SHELL -x $testScriptPath 2>&1 | tee $out/build.log
      ${pkgs.darwin.cctools}/bin/otool -l test> $out/load_commands
    '';
  };
in
  pkgs.runCommand "all-tests" {
    passthru = lib.mapAttrs buildTest tests;
  } (
    ''
      mkdir $out
    '' +
    lib.concatStringsSep "\n" (
      lib.mapAttrsToList (name: script: ''
        ln -s ${buildTest name script} $out/${name}
      '') tests
    ) + ''
      cd $out
      {
        showCommand() {
          local cmdName=$1
          echo -ne "\t$cmdName: "
          awk "/$cmdName/ { show = 1 }; /Load command/ { show = 0 }; show" < $i | tr '\n' '\t'
          echo
        }

        for i in */load_commands; do
          echo $i
          showCommand LC_BUILD_VERSION
          showCommand LC_UUID
          showCommand LC_VERSION_MIN_MACOSX
        done

        echo

        echo "Linker warnings:"
        grep -nH 'warning' */build.log || true
      } > summary 2>&1
    ''
  )

---

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[holymonson]

FYI, I was working on #111658 which could support multiple versions of sdk (by generating several info lists). I'm not sure if you need it, but I will pause here.

[Ericson2314]

Suggested change

	postPatch = lib.optionalString stdenv.hostPlatform.isDarwin ''
	substituteInPlace src/luarocks/core/cfg.lua --subst-var-by 'darwinMinVersion' '${stdenv.hostPlatform.darwinMinVersion}'
	postPatch = lib.optionalString stdenv.targetPlatform.isDarwin ''
	substituteInPlace src/luarocks/core/cfg.lua --subst-var-by 'darwinMinVersion' '${stdenv.targetPlatform.darwinMinVersion}'

because we should do this for other -> darwin cross too, I believe?

[thefloweringash]

I don't know if nixpkgs has has working cross compilation for lua packages, but assuming it does, targetPackages seems correct here 👍.

[thefloweringash]

With the debugging commit the evaluation error is visible:

darwinPlatform: unknown platform for x86_64-unknown-linux-gnu

Is there an idiom for platform configuration that doesn't apply to all platforms?

[thefloweringash]

This ensures MACOSX_VERSION_MIN is set for the correct platform, but now I'm finding a significant downside. As is, x86_64-darwin uses clang 7 and aarch64-darwin uses clang 11. When cross-compiling for aarch64-darwin, MACOSX_DEPLOYMENT_TARGET=11.0. When clang 7 builds for the build platform it uses this environment variable. It's incorrect since it's for the wrong platform, and it's also an error since clang prior to 11 rejects values not starting with 10. It's easy to work around with NIX_CFLAGS_COMPILE_FOR_BUILD in a given package, but I'm considering adding -mmacos-version-min to cflags in the cc-wrapper.

[thefloweringash]

This is ready for review. Still outstanding: how to represent the absence of value for darwin* flags on platforms that aren't darwin.

[veprbl]

@thefloweringash This does not evaluate btw.

[thefloweringash]

@thefloweringash This does not evaluate btw.

I didn't know a nice way to have the darwinPlatform be absent on platforms other than darwin. I had it as a throw which was triggering on linux platforms. I've made it a null, which doesn't quite seem correct but should at least not break unless it's actually used, and not silently fail if interpolated.

[Ericson2314]

null seems fine to me.

[Ericson2314]

@thefloweringash Are you going to change https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/darwin/xcode/sdk-pkgs.nix later then?

Also, was the MACOSX_DEPLOYMENT_TARGET issues resolved? This sounds like the sort of thing where we could do MACOSX_DEPLOYMENT_TARGET and MACOSX_DEPLOYMENT_TARGET_FOR_BUILD, and the build compiler would the "suffix salt" black magic to get the right one. (c.f. the pkg-config wrapper and PKG_CONFIG_PATH for the most self-contained example of that.)

[Ericson2314]

Is this ready now?

[thefloweringash]

@thefloweringash Are you going to change https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/darwin/xcode/sdk-pkgs.nix later then?

Updated to include this. I don't know of any example that uses it in the nixpkgs tree so I haven't tested it. If anyone that relies on cross compiling to iPhone has a chance to test it, it would be appreciated.

Also, was the MACOSX_DEPLOYMENT_TARGET issues resolved? This sounds like the sort of thing where we could do MACOSX_DEPLOYMENT_TARGET and MACOSX_DEPLOYMENT_TARGET_FOR_BUILD, and the build compiler would the "suffix salt" black magic to get the right one. (c.f. the pkg-config wrapper and PKG_CONFIG_PATH for the most self-contained example of that.)

I've now added support for the MACOSX_DEPLOYMENT_TARGET_FOR_BUILD variable, which required making a variable mangler that operated on a singular value.

Is this ready now?

I believe so.

[domenkozar]

@Ericson2314 feel free to merge if this looks good to you.

[Ericson2314]

Took me a second to figure out what mangleVarSingle is for, but yeah looks good. Sometimes I wish we had build-cc, cc, and target-cc rather than these tools prefixed with concrete platforms, and then we wouldn't have to worry about conflicts. Thanks for handling the conflict case correctly.

[Ericson2314]

(As to iOS cross, I will someday need to bump our stuff to use this version of Nixpkgs, but i don't have the phones to test with me. I like this design better so if there are regressions they should be plenty easy to fix and totally worth it :).)