Tracking issue: Update Darwin bootstrap files

[stephank]

I'm opening a tracking issue for updating the Darwin bootstrap files to document pending changes, and to see if/when we have enough reasons to perform an update.

The last update was in 2021 as part of #105026. I've been told updates to bootstrap-files are supposed to be rare, and there should be good rationale. Once we have that, plus some thumbs up from @NixOS/darwin-maintainers, we should be able to find help getting tarballs.nixos.org updated via #infra:nixos.org.

Currently pending changes:

• stdenvBootstrapTools: native aarch64-darwin build #202347
  This PR changed the structure of the bootstrap files. Currently, there is the main bootstrap-tools archive, plus some separate executables required to unpack it (sh, bzip2, mkdir, cpio). The new structure is just an archive, and uses the builtin Nix support for .nar.xz unpacking. The PR contains more rationale, plus there's an example commit.

• apple-m1 compiler optimisations for aarch64-darwin? #166401
  Maybe(?) update LLVM and change arch flags for Apple Silicon.

• Anything else?

Reasons to delay:

• With only the first one above currently on master, the pending changes do not significantly affect the resulting stdenv.

Reasons to move forward:

• Maybe we want to resolve the discrepancy in structure (between the bootstrap-tools we build and the bootstrap-tools we fetch) before 23.05?

[reckenrode]

One reason to update bootstrap tools is it allows the x86_64-darwin stdenv to be bootstrapped with a newer SDK since that needs pbxz and the tbd tools. That was a blocker to moving forward with #180931, which replaced the override-based approach with a fully bootstrapped 11.0 stdenv on x86_64-darwin.

I’ve also started experimenting with exposing an sdkPkgs attribute that would allow you to use packages built against the newer SDK without having to stuff them into the callPackage workaround. That would also depend on newer bootstrap tools for the same reason.

Update: I should add that #180931 needs revised to break out the SDK changes into its own PR since many of the packages have been updated separately.

[wegank]

I believe almost all the packages in #180931 now build on x86_64-darwin, or at least at some point.

In the meantime, it is currently difficult to build GTK-dependent packages with darwin.apple_sdk_11_0.callPackage without hacks, simply because of the propagatedBuildInputs here:

nixpkgs/pkgs/development/libraries/gtk/3.x.nix

Lines 151 to 152 in 6566705

	Cocoa
	QuartzCore

And I think bumping everything to 11.0 is the most direct way to solve this...

[reckenrode]

I believe almost all the packages in #180931 now build on x86_64-darwin, or at least at some point.

I wouldn’t be surprised. I lost momentum when I got blocked on bootstrap-tools stuff. Doing them separately was probably the right approach anyway.

In the meantime, it is currently difficult to build GTK-dependent packages with darwin.apple_sdk_11_0.callPackage without hacks, simply because of the propagatedBuildInputs here:

MoltenVK has a similar problem. It can’t currently be used with any library that propagates SDK-based build inputs (like libGLU).

And I think bumping everything to 11.0 is the most direct way to solve this...

The reason why I’m exploring exposing pkgsDarwinSDK.v11_0 (or something to that effect) is to continue supporting users on older versions of macOS and the source-based SDK. I’d also like to see at least the 13.0 SDK for MoltenVK because of the added features Metal 3 enables, but bumping stdenv to 13.0 now isn’t really reasonable.

[lovesegfault]

I can upload new bootstrap tarballs if needed.

[stephank]

One reason to update bootstrap tools is it allows the x86_64-darwin stdenv to be bootstrapped with a newer SDK since that needs pbxz and the tbd tools. That was a blocker to moving forward with #180931, which replaced the override-based approach with a fully bootstrapped 11.0 stdenv on x86_64-darwin.

I'm not sure I follow. By 'override-based approach', do you mean the callPackage solution we have now? Both 10.12 and 11.0 SDKs already use pbzx? Isn't a 11.0 stdenv undesirable at this point, or is the plan to bootstrap a 10.12 SDK with a 11.0-based bootstrap tools?

I can upload new bootstrap tarballs if needed.

Thanks for the offer. I deliberately didn't ping, because I expected some discussion first, but can ping when there is a decision, if you'd rather not stay subscribed. 🙂

[reckenrode]

I'm not sure I follow. By 'override-based approach', do you mean the callPackage solution we have now? Both 10.12 and 11.0 SDKs already use pbzx? Isn't a 11.0 stdenv undesirable at this point, or is the plan to bootstrap a 10.12 SDK with a 11.0-based bootstrap tools?

I’m referring to the callPackage solution we have now. It relies on being able to override cc, which just happens to work for clang. While working on #180931, I explored bootstrapping the stdenv at @toonn’s suggestion. The direction it was going was to have a private 11.0 pkgs that the callPackage could selectively reference.

The callPackage approach doesn’t compose very well. I’ve been looking at using the 11.0 SDK to build Wine, but it’s going to be pretty messy. I think Qt 5 is another example of a package that’s too complex to build with just the override. It would be much easier if one could just reference the 11.0 SDK pkgs directly.

I have a proof of concept working. It builds Wine just by referencing <11.0 pkgs>.wine64Packages.wine. It’s a bit hacky though and requires the updated bootstrap-tools. From what I understand, while the PR was merged to add pbzx and tbd to x86_64-darwin, the bootstrap tools archive hasn’t been updated. Using a bootstrapped 11.0 SDK stdenv on x86_64-darwin has the same requirements as aarch64-darwin, so it needs the same tools.

As far as whether it’s desirable to build an 11.0 stdenv and expose a pkgs, I don’t know. I think it’s a conversation worth having (though maybe not in this issue). From the perspective of MoltenVK, it’s going to increasingly make use of features introduced in Metal 3, which requires the 13.0 SDK. It can gracefully fall back on older systems, but it can’t use newer features without using the newer SDK.

Hopefully that helps clarify. If not, please let me know.

[stephank]

@reckenrode I think you explained it well, thanks, and I won't dive in too much further here. Some specific questions for this issue:

• What further updates to bootstrap-tools would help along experimentation in those areas, and which are not yet on master?
  • I understand the pbzx stuff is already in, and it just needs an upload?
  • Are updates to Clang or the SDK components we package inside bootstrap-tools helpful in any way? (Not sure what the minimum Clang is to bootstrap a 13.0 SDK stdenv, for example. But maybe we simply don't know yet.)
• Is there any benefit to doing this within the 23.05 release window?

[reckenrode]

My apologies for the slow reply. I meant to reply earlier but got sidetracked.

• What further updates to bootstrap-tools would help along experimentation in those areas, and which are not yet on master?

The current state of master should be sufficient. What I need is parity between aarch64-darwin and x86_64-darwin because I’m trying to build the stdenv on x86_64-darwin using the same SDK that’s used on aarch64-darwin.

One thing I’d like to explore is using the linker in LLVM instead of cctools (since Apple seems to have stopped supplying source updates), but I wouldn’t add it to the bootstrap-tools proactively.

• I understand the pbzx stuff is already in, and it just needs an upload?

Correct.

• Are updates to Clang or the SDK components we package inside bootstrap-tools helpful in any way? (Not sure what the minimum Clang is to bootstrap a 13.0 SDK stdenv, for example. But maybe we simply don't know yet.)

That would also result in a bump in the clang used in the stdenv in the regular SDKs?

You’re right that we don’t know what clang is needed to bootstrap the 13.0 SDK. I wouldn’t proactively bump clang just for that, though there may be other reasons to do it (that don’t come to my mind but others might have).

• Is there any benefit to doing this within the 23.05 release window?

The separate bootstrapped stdenv stuff seems unlikely for 23.05, so I wouldn’t try to make the release window just for that. I also think we’ve usually held off making changes this close to the release window in the past, so it might be something to target for 23.11.

[reckenrode]

While working on an LLVM bump to LLVM 15 (in support of #229210), I ran into the following issues. They can be worked around, and I have, but it’s preferable if the bootstrap tools were updated to make the workarounds unneeded.

• sigtools needs updated to 0.1.3. LLVM 15 assumes it can call codesign --sign, but that’s not supported by the version in the bootstrap tools. Building compiler-rt fails due to missing support for --sign.
• sigtools on x86_64-darwin. LLVM 15 assumes that the version of ld64 we have requires manual ad hoc signing regardless of platform. Building compiler-rt fails without codesign.
• CoreFoundation has definitions in CFAvailability.h that cause the build to fail. I believe this is related to elaborated-enum-base incompatible with NS_ENUM llvm/llvm-project#48757. Passing the flag to suppress the error does work (for now anyway).

Those are all I have found so far that have been blockers. My modifications for the bump allow it to use a different bootstrap compiler from the final one. It’s also written to play nicely with LLVM 11 even after #166401 is addressed.

[reckenrode]

After doing some more work, I don’t think any of the changes above are needed. I reworked the stdenv bootstrap to (re)build what it needs in stage 1. It may not even be necessary to include rewrite-tbd or print-exports on x86_64-darwin for newer SDKs, but I haven’t tried doing that (yet?).

Regarding the two bullets in the issue:

I also handle #166401 in the bootstrap. If the clang is bootstrap, it assumes the clang version is older than 12 and overrides the build flags. For clang in nixpkgs, it uses the flags defined in platform.nix unless your clang is 11, then it uses a compatible cpu definition.

I don’t think the changes to nar.xz will impact what I’m doing as long as the unpack scripts are updated accordingly. The reworked stdenv bootstrap tries to assume as little as possible about what it gets from bootstrap tools. The goal is to decouple from it, so bumps to LLVM don’t also require bumps to the bootstrap tools.

[reckenrode]

With the stdenv rework merged, I can confirm no changes or additions are required from the bootstrap tools. I think the place for the platform optimization stuff is in cc-wrapper like it’s done for GCC. Doing it in the stdenv is too brittle and not the correct place anyway.

[toonn]

Should this move forward again? We've waited quite a while for updates to group. Maybe the LLVM bump @reckenrode is working on?

[reckenrode]

It’s not needed for the LLVM update, so it should probably wait until after #241692 is merged to minimize the risk of breaking something and blocking the PR.

[viraptor]

@reckenrode wrong PR link?

[reckenrode]

@reckenrode wrong PR link?

Thanks, fixed it. I messed up transcribing it while replying on my phone.

[ghost]

I'm working on a couple of changes to update the bootstrap tools. I made some modifications to the llvm toolchain for versions 12+ and thus would like to drop llvm11. also, stdenvBuildTools is now broken on staging-next and unpinning llvm11 will fix.

• freshBootstrapTools: remove overlay, prune tools, use tar.xz archive #295557
• stdenv.darwin: bootstrap darwin using updated tools #295558

So far i've only tested on x64 darwin with the sandbox enabled.

things i've done

1. unpin llvm11 and remove override so the tools will be built with current stdenv
2. add @loader_path/. to various libs that reference other libs. eg: libc++ references and re-exports libc++abi
3. add a couple checks to just validate that shared objects with an @rpath contain the library in lib
4. use getDev, getLib, getBin when copying out of the packages. hopefully this minimizes breakage in the future
5. somewhat aligned the darwin make-bootstrap-tools.nix with the linux side
6. remove unneeded LLVM libs and tools to reduce overall size by more than 1/2. compressed tarball is about 40M

Tested on aarch64 and x64 darwin

thx

[ghost]

closed by #302387

[samueldr]

Closing, assumed indeed closed by #302387.

If this is inaccurate, please re-open and (or) add a comment detailing why.