darwin bootstrap-tools bump LLVM to 11 #121055

toonn · 2021-04-28T17:05:08Z

Motivation for this change

To bump the Apple SDK we need an updated bootstrap-tools for the Darwin stdenv.

Current status

I have added two tags on my branch. bs1 points to the latest commit that can be built with the current bootstrap-tools, for everything after this you have to pass in this bootstrap-tools, and we need another bootstrap-tools to get darwin.Csu a dependency of stdenv to build, this would be tagged bs2. current points to the commit I'm currently working on, something's depending on a libunwind path it shouldn't be when trying to make-bootstrap-tools. The commits dealing with darwin.xnu and darwin-stubs are specific to the Apple SDK bump. I have a messy build log of this, which may help (or hinder) understanding the changes.

One big change in this branch is the bump from LLVM 7 to 11. This is because of this build failure. I didn't really see a way forward and another part of darwin.CF was problematic because of LLVM 7 anyway so @LnL7 figured I might as well try bumping stdenv (and bootstrap-tools) LLVM, this was based on previous work trying to bump it to 9 #85151. If the priority is to bump only the Apple SDK (and we find an alternative solution to that build failure) it might be less time consuming to delay the LLVM bump and continue on my apple_sdk-bump branch.

cc @domenkozar

pkgs/os-specific/darwin/darwin-stubs/default.nix

pkgs/os-specific/darwin/swift-corelibs/corefoundation.nix

pkgs/os-specific/darwin/apple-source-releases/xnu/3789_70_16/default.nix

nixos-discourse · 2021-05-03T12:54:47Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/2

pkgs/stdenv/darwin/make-bootstrap-tools.nix

toonn · 2021-06-01T20:38:32Z

Ok, so the latest force-push is the branch with all the changes bumping from LLVM 9 to 11 for both the Darwin bootstrap-tools and the stdenv.
To generate the bootstrap-tools for the tip of this branch (tag bs2) you need to first build an intermediate bootstrap-tools at tag bs1.

Commit 7db1708e adds an argument to make-bootstrap-tools.nix that can be used to do so. I suppose this isn't useful for hydra, we'll probably have to have it build a bootstrap at bs1 and upload that somewhere and add a commit changing the url in stdenv?

For development I usually do the following:

> git checkout bs1
> nix-build pkgs/stdenv/darwin/make-bootstrap-tools.nix -A build
/nix/store/amq6frjddx18mmycj1ng0kc2r8rk4rbg-stdenv-bootstrap-tools
> git checkout bs2
> nix-build --no-out-link -E 'let np = import <nixpkgs> {}; bts = _: tool: "/nix/store/amq6frjddx18mmycj1ng0kc2r8rk4rbg-stdenv-bootstrap-tools/on-server/" + tool; bootstrapFiles = { bootstrapFiles = np.lib.mapAttrs bts { bzip2 = "bzip2"; cpio = "cpio"; mkdir = "mkdir"; sh = "sh"; tarball = "bootstrap-tools.cpio.bz2"; }; }; cfg = { stdenvStages = args: let args_ = args // bootstrapFiles; in (import pkgs/stdenv/darwin args_).stagesDarwin; }; make-bootstrap-tools = import pkgs/stdenv/darwin/make-bootstrap-tools.nix bootstrapFiles; pkgs = import ./. cfg; in make-bootstrap-tools.build'
/nix/store/lmpwkir5gjwbkcjna30y4rraibwmq1x5-stdenv-bootstrap-tools

This relies on two commits since merging the Apple Silicon PR #105026, 2987d66 and 3820773. This may be relevant if we target staging. The latter wasn't present there last I checked.

domenkozar · 2021-06-02T16:00:42Z

@toonn if this is ready for review, mark this PR as not draft (there are failing eval checks).

toonn · 2021-06-02T19:46:03Z

@domenkozar, this PR is still a draft. There's probably package breakages we need to fix, working with @LnL7 to get a hydra jobset for this.

I'm a little stumped by the ofborg-eval error tbh. I didn't touch llvm-manpages, it's still defined for LLVM 11 and it's still in allpackages.nix. I'm not sure what the attribute being missing can mean at this point.

domenkozar · 2021-06-02T20:47:53Z

That error will be fixed with #125184, hope to merge that tomorrow

toonn · 2021-06-02T20:52:19Z

@domenkozar, ah, thanks. I had the weirdest case of déjá vu because of the "recommonmark." It's because I saw that PR earlier.

I was having a hard time testing new bootstrapFiles because `make-bootstrap-tools.nix` imports `pkgspath` but does not pass anything but the current system. This is merely for convenience and I'm not entirely certain it's a sensible thing to do, maybe generating new bootstrapFiles while overriding the current bootstrapFiles isn't something you're supposed to do?

@LnL7

Credits to @LnL7, this is based on commit ae68f1c from PR NixOS#85151.

We need to include the library with the version suffix because `libclang-cpp.dylib` is just a symlink.

pkgs/stdenv/darwin/make-bootstrap-tools.nix

LnL7 · 2021-06-06T17:09:30Z

pkgs/os-specific/darwin/apple-source-releases/Libsystem/default.nix

-    #define TARGET_OS_IPHONE        0
-    #define TARGET_IPHONE_SIMULATOR 0
-    #define TARGET_OS_LINUX         0
+    #define TARGET_OS_MAC               1


Shouldn't be needed for the initial bootstrap build yet, removing the stdenv rebuild.

You're right, forgot to remove it when adding it in make-bootstrap-tools.nix. >.<

I'm still building the intermediate bootstrap-tools to verify that the preFixup hook actually works as expected btw.

Overridden installPhase did the trick.

Should I go ahead and mark this ready for review?

`TargetConditionals.h` was missing several definitions, like `TARGET_OS_TV` that are part of SDK 10.12 at least. And one that doesn't seem to occur in any SDK afaict, `TARGET_OS_EMBEDDED_OTHER`. I added the definitions from SDK 10.12 verbatim and defined `TARGET_OS_EMBEDDED_OTHER` to be equal to `0`. This is a modified version of a patch to avoid a stdenv rebuild.

Changes to llvmPackages have caused the `libclang-cpp*.dylib` files to be included in the `clang-unwrapped.lib` output. So we no longer need to copy them from libclang.

sternenseemann · 2021-06-09T10:11:46Z

What happened to the global llvmPackages bump?

toonn · 2021-06-09T10:34:19Z

@sternenseemann, that's upcoming. We need this intermediate bootstrap-tools built first.

toonn · 2021-06-09T22:29:52Z

@sternenseemann, continuing in #126411.

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Apr 28, 2021

github-actions bot added the 6.topic: stdenv Standard environment label Apr 28, 2021

toonn force-pushed the bootstrap-tools-bump branch 2 times, most recently from 6ebf3e0 to 423a47a Compare April 28, 2021 17:19

SuperSandro2000 reviewed Apr 30, 2021

View reviewed changes

pkgs/os-specific/darwin/darwin-stubs/default.nix Outdated Show resolved Hide resolved

pkgs/os-specific/darwin/swift-corelibs/corefoundation.nix Outdated Show resolved Hide resolved

pkgs/os-specific/darwin/apple-source-releases/xnu/3789_70_16/default.nix Outdated Show resolved Hide resolved

domenkozar mentioned this pull request Apr 30, 2021

Native support for Apple Silicon #105026

Merged

21 tasks

toonn force-pushed the bootstrap-tools-bump branch 3 times, most recently from 0b983d3 to 027d3fa Compare April 30, 2021 21:46

ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 30, 2021

toonn force-pushed the bootstrap-tools-bump branch from 027d3fa to 152817e Compare May 16, 2021 16:14

sternenseemann reviewed May 16, 2021

View reviewed changes

pkgs/stdenv/darwin/make-bootstrap-tools.nix Outdated Show resolved Hide resolved

toonn force-pushed the bootstrap-tools-bump branch from 152817e to 9c6795e Compare May 17, 2021 12:18

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label May 17, 2021

toonn force-pushed the bootstrap-tools-bump branch 3 times, most recently from 53c0636 to 4f1204a Compare May 23, 2021 17:08

toonn force-pushed the bootstrap-tools-bump branch 2 times, most recently from a9046ab to a95f566 Compare June 1, 2021 20:17

toonn changed the base branch from staging to master June 1, 2021 20:20

toonn changed the title ~~darwin stdenv bootstrap-tools bump~~ darwin stdenv and bootstrap-tools bump LLVM to 11 Jun 1, 2021

ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 8.has: clean-up 8.has: package (new) This PR adds a new package labels Jun 2, 2021

toonn added 4 commits June 5, 2021 19:56

bootstrap-tools: Update to LLVM 11

4a26982

bootstrap-tools: Include libclang-cpp.dylib

d259fcc

Credits to @LnL7, this is based on commit ae68f1c from PR NixOS#85151.

bootstrap-tools: Include libclang-cpp.11.1.dylib too

fa49e70

We need to include the library with the version suffix because `libclang-cpp.dylib` is just a symlink.

toonn force-pushed the bootstrap-tools-bump branch 2 times, most recently from b5f8349 to 693f353 Compare June 5, 2021 18:41

ofborg bot requested review from copumpkin and gridaphobe June 5, 2021 18:55

ofborg bot added 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux and removed 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Jun 5, 2021

LnL7 reviewed Jun 6, 2021

View reviewed changes

toonn force-pushed the bootstrap-tools-bump branch from 693f353 to cad0245 Compare June 6, 2021 17:26

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin and removed 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ labels Jun 6, 2021

toonn added 2 commits June 6, 2021 20:17

bootstrap-tools: libclang-cpp already included

c7b433a

Changes to llvmPackages have caused the `libclang-cpp*.dylib` files to be included in the `clang-unwrapped.lib` output. So we no longer need to copy them from libclang.

toonn force-pushed the bootstrap-tools-bump branch from cad0245 to c7b433a Compare June 6, 2021 18:22

toonn changed the title ~~darwin stdenv and bootstrap-tools bump LLVM to 11~~ darwin bootstrap-tools bump LLVM to 11 Jun 6, 2021

domenkozar mentioned this pull request Jun 7, 2021

pythonPackages.watchdog 2.0.0 is broken on darwin #113777

Closed

toonn marked this pull request as ready for review June 8, 2021 20:27

LnL7 approved these changes Jun 9, 2021

View reviewed changes

LnL7 merged commit bdffd0b into NixOS:master Jun 9, 2021

toonn mentioned this pull request Jun 9, 2021

Darwin stdenv bump LLVM to 11 #126411

Merged

thefloweringash mentioned this pull request Jul 10, 2021

cpython: disable no-semantic-interposition on darwin #129669

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

darwin bootstrap-tools bump LLVM to 11 #121055

darwin bootstrap-tools bump LLVM to 11 #121055

toonn commented Apr 28, 2021

nixos-discourse commented May 3, 2021

toonn commented Jun 1, 2021 •

edited

Loading

domenkozar commented Jun 2, 2021 •

edited

Loading

toonn commented Jun 2, 2021

domenkozar commented Jun 2, 2021

toonn commented Jun 2, 2021

LnL7 Jun 6, 2021

toonn Jun 6, 2021

toonn Jun 6, 2021

toonn Jun 7, 2021

toonn Jun 7, 2021

sternenseemann commented Jun 9, 2021

toonn commented Jun 9, 2021

toonn commented Jun 9, 2021

darwin bootstrap-tools bump LLVM to 11 #121055

darwin bootstrap-tools bump LLVM to 11 #121055

Conversation

toonn commented Apr 28, 2021

Motivation for this change

Current status

nixos-discourse commented May 3, 2021

toonn commented Jun 1, 2021 • edited Loading

domenkozar commented Jun 2, 2021 • edited Loading

toonn commented Jun 2, 2021

domenkozar commented Jun 2, 2021

toonn commented Jun 2, 2021

LnL7 Jun 6, 2021

Choose a reason for hiding this comment

toonn Jun 6, 2021

Choose a reason for hiding this comment

toonn Jun 6, 2021

Choose a reason for hiding this comment

toonn Jun 7, 2021

Choose a reason for hiding this comment

toonn Jun 7, 2021

Choose a reason for hiding this comment

sternenseemann commented Jun 9, 2021

toonn commented Jun 9, 2021

toonn commented Jun 9, 2021

toonn commented Jun 1, 2021 •

edited

Loading

domenkozar commented Jun 2, 2021 •

edited

Loading