Discussion: The future of SDKs on Darwin #242666
Comments
|
I’m replying with my perspective separately to keep it from biasing the OP. I would like to see a singular approach to overriding the SDK and a pattern that plays nicely with it. It seems to me that the SDK is analogous to the libc, and so should live as an attribute on the stdenv. When you need an SDK package, you should reference
For example, given an adapter might like: overrideSDK = stdenv: new_sdk:
let mkCC = cc:
cc.override {
bintools =s tdenv.cc.bintools.override { libc = new_sdk.Libsystem; };
libc = new_sdk.Libsystem;
};
in
(overrideCC stdenv (mkCC stdenv.cc)).override {
extraBuildInputs = [ new_sdk.CF ];
hostPlatform = stdenv.hostPlatform // {
darwinSdkVersion = new_sdk.version;
};
} // {
apple_sdk = new_sdk;
};Given the following derivation: { stdenv, zlib }:
stdenv.mkDerivation {
name = "my-package";
version = "1.0.0";
buildInputs = [ zlib ]
++ lib.optionals stdenv.isDarwin (with stdenv.apple_sdk.frameworks; [ CoreFoundation ]);
}There are two ways to use this. With an explicit my-package = callPackage ./path/to/my-package {
stdenv = if stdenv.isDarwin then overrideSDK stdenv darwin.apple_sdk_13 else stdenv;
};Auto-called inside the derivation: { stdenv, darwin, overrideSDK, zlib }:
let
stdenv' = if stdenv.isDarwin then overrideSDK stdenv darwin.apple_sdk_latest else stdenv;
in
stdenv'.mkDerivation {
pname = "my-package";
buildInputs = [ zlib ]
++ lib.optionals stdenv'.isDarwin (with stdenv'.apple_sdk.frameworks; [ CoreFoundation ]);
# etc
}Which pattern is preferred should follow how different stdenv requirements are handled after the simple package paths RFC is implemented. |
|
Regarding SDK evolution, I would like to see the 10.12 SDK retrofitted into the structure being implemented #229210. The source SDK components should be implemented as overlays. I think being able to incrementally convert an SDK will make it easier to move forward on that front. We should also be aiming for an ideal of “source-based headers plus tbd files” for purity. Trying to build Apple’s OSS stuff can be pretty painful, especially newer releases. For non-SDK applications (e.g., adv_cmds, system_cmds, etc), though should be treated as normal packages and updated to be as new as will build on the default SDK for a platform. When they are common packages that are already maintained in nixpkgs, they should be dropped in favor of the nixpkgs versions (e.g., ICU and libiconv). |
reckenrode
changed the title
|
Since I agree with most of this, I don't really have much to add. I don't think we need to worry about 10.13, that work is progressing, I will be opening PRs soon. This might be out of scope but I want to bring it up anyway. To me the ideal scenario would be packages simply specify what minimum macOS version they support; potentially a maximum too but that's pretty much a non-issue at this point. The Nixpkgs infrastructure would then provide whatever this minimum version is, probably preferring source releases over the SDK distributed by Apple. Now, while this minimum version information seems appropriate for a package's Maybe Is it feasible for us to take something like this into account? In the simple case it'd mean maintainers just look up what the minimum supported macOS version is upstream, pass it in however we decide on doing it and be done with it. In the probably more common case it'd mean looking up what version of macOS symbols were introduced in. This could be scripted to reduce the pain though. |
Will that introduce a new |
It seems in scope since it pertains to how the SDK interacts with nixpkgs.
Is that the minimum runtime or build requirement? MoltenVK supports back to 10.11, but it really should be built with the latest SDK. It can’t even be built with the 10.11 (or 10.12) SDKs, and building with pre-latest SDKs negatively impacts feature availability when run on newer macOS releases.
How would it interact with language frameworks and alternate stdenvs? Would the SDK still have to vend e.g.,
The script sounds handy regardless. It would also be nice to know if some of those symbols are weakly linked (because it might allow well-behaved applications to be built with a newer SDK with an older deployment target). |
Tangent to the main point here (happy to rm if it's a distraction): I guess this implies some thinking/coordination with installers and about what should happen on any existing installs under this floor? (I have no clue if there actually are any left in the wild...)
|
No, I think it’s something that needs consideration too. If Darwin is going to regularly add SDKs and update LLVM, we need to determine what the deprecation process looks like for packages and platforms and document it.
I assume coordinating with the installers would be part of the deprecation process. What channel do they use by default? That would also affect things since the unstable channel would change over before the stable one.
It would be nice if there were a place in the release notes for Darwin-related changes, especially deprecations and incompatibilities. Given an LLVM 18 bump won’t happen until this time next year for the 24.11 release, an announcement in 23.11 would provide a year’s notice of the upcoming change. |
The official installer uses unstable. I don't understand what exactly the detsys installer does here. They do cite what they don't do:
But I haven't gone through it to understand the full implications. |
That’s what I thought, so we have less time than until the 24.11 release. Can we get a deprecation notice or some changes in the installer to let users know that 10.12 support will be dropping next year? If necessary we might be able to push it out to 2025 by reverting the commit from libc++ that drops 10.12 support, but we’d also need to make sure it’s not using 10.13 APIs anywhere else (and the LLVM maintainers are willing to carry the patch).
I assume the lack of channels is because they assume people will be using flakes, but I reached out on Matrix to see if anyone had any experience with it to confirm. |
|
Another topic worth discussing is how to handle open source releases that are out of date or no longer available. I ran into this while testing the clang 16 update. Ruby fails to build with clang 16 on x86_64-darwin because it needs I’m currently planning to use the 11.0 SDK to build Ruby, but it would be better if there were a way to provide the up-to-date headers for the 10.12 SDK. |
|
I think the plan to overlay source releases on the SDKs kinda solves the outdated/unavailable source problem. If there's no source for something, we get it from the SDK, like XPC for example. Unless an open source reimplementation comes along. |
|
I think SDKs and platforms are strongly related, is there some way how we can align the darwin sdks with the concept of nix platforms? For example the platforms in We might need to represent the distinction between the host and target platform: if I understand the MoltenVK case correctly for example, the host platform needs the latest SDK, but the target platform can be as low as |
|
Because we don't use source SDKs in all cases in practice we can target macOS without nix from nixpkgs easily at the moment, by using Is this a use case we want to support in nixpkgs, or something we should defer to external solutions such as https://github.com/DavidEGrayson/nixcrpkgs/? |
Would it be possible to mix packages from different platforms under that scheme?
What would that look like in a derivation? Something like this? let
stdenv' = stdenv.override {
targetPlatform = stdenv.targetPlatform // { darwinMinVersion = "13.3"; };
hostPlatform = stdenv.hostPlatform // { darwinMinVersion = "10.11"; };
};
in
stdenv.mkDerivation {
depsBuildTarget = [ /* ??? */ ];
/* MoltenVK stuff */
} |
The source SDK ought to be able to do that too, but work needs to be done to make it possible.
Ideally, we should not need to defer to external projects. It seems like something we can support and should. |
|
#244471 is an example of where slight differences in the source SDK caused an issue. Foundation should reexport libobjc, but it did not. I opened NixOS/darwin-stubs#10 regarding the issue. If |
It should be, the same way we can currently build container images on darwin by using a linux package set. Not sure if that pattern fits the use cases you have in mind but at least it's possible, and we can add helper functions around it to make it more ergonomic.
Yeah, something like that. I'm not sure about the specifics, it looks like they're the wrong way around, but maybe if we add |
Here is an example of what I have in mind: Wine is built with the 10.12 SDK. It depends on MoltenVK, which is built with the 11.0 SDK. MoltenVK is a
As I understand it, the host is what uses the dylibs, which for MoltenVK can be as old as 10.11. The build environment would need to be 13.3. I went with Regarding Currently, not many packages bother to change the minimum version. mongodb-6_0 does (setting it to 10.14), and I have a patch in my queue to have pybind11 set it to 10.13 to silence (arguably bogus) checks clang 16 does when using aligned allocations. It’s currently set on x86_64-darwin to 10.12 to reflect the fact that’s the default SDK and target version. |
That should be ok if whichever package uses the non-default SDK does so explicitly in the package derivation, like the However what does that mean for a consumer of those packages? Does that mean that changing the default SDK at the package set level would only apply to certain packages? Hopefully packages using a specific non-default SDK would be the exception, and once we have many SDKs, we can have package specify the minimum SDK version, and then they can use either that, or the default SDK version, if the default SDK version is new enough. That would let consumers override both the host and target platforms to be newer than the current nixpkgs default, and have the packages pick it up. Example of which SDK would be used to build packages:
|
I’d like to work through the MoltenVK to make sure I understand how this would work (especially since I do now realize I had it backwards regarding host and target). Assume that That’s very similar to the status quo today. The 11.0 SDK stdenv sets The other question I have is how those dependencies are determined. Would they be spliced into the derivation? You just use |
|
I am used to MacPorts I am not certain that the last assumption in your original post can be done. Macports works and can provide current builds going back to Tiger (10.4) in some cases. The limit tends to be what upstream does. What they do is for each different version of OSX/macOS they use a different version of Xcode and Apple libraries including clang. For most software the build instructions are the same independent of the macOS version but they can be changed so that new Apple APIs can be used on newer macOS versions. In general this makes sense to me as each version of macOS has a different stdenv. |
Why not? That’s how things work on Darwin normally outside of nixpkgs. An application can link a framework regardless of which SDK was used to build it as long as any common dependencies are shared (e.g., they link the same zlib, libxml2, etc). MoltenVK is my go-to example because it supports 10.11 at runtime even though it cannot be built with older SDKs. It currently builds with the 11.0 SDK, but it really should be built with the 13.3 one. Xcode 11 support was recently dropped from MoltenVK. It’s only a matter of time before it no longer builds with any SDK in nixpkgs (without #229210).
I like MacPorts, but that approach isn’t feasible for nixpkgs. There already are not enough Darwin builders, and many Linux maintainers have no access to Darwin hardware. If they had to make sure packages built across a matrix of platforms, I fear even fewer would bother. The pool of active Darwin maintainers is also pretty small. I expect it’d be a recipe for bitrot. 😕
The only things used from the system are the libc and system frameworks (because there is no other way). The system libc++ is not used nor is Apple’s toolchain (except¹ for cctools and ld64, which are built from source). nixpkgs builds LLVM from upstream (not Apple’s fork but upstream LLVM) and uses that with the SDK headers and stubs to build packages. Theoretically, this setup should allow cross-compilation to Darwin. It’s not there yet. Currently, only x86_64-darwin to aarch64-darwin works (and static compilation on aarch64-darwin). ¹ I started replacing cctools with equivalents from LLVM as part of the Darwin stdenv rework. Eventually, it might be possible to drop Apple’s tools and make Darwin a |
I don't understand why target and host would ever be different here. That should only be relevant for things like building the cross-compiler itself from Darwin to some other platform and that's not a thing yet. Am I missing something? I really like the idea of making the SDK part of the platform; it feels like the obvious place to put it. Overriding it should then only be a matter of overriding As for dependencies which propagate a specific SDK version (i.e. libGLU), perhaps stdenv.mkDerivation could automatically set the hostPlatform's SDK to that propagated SDK version (throwing an error if there are multiple different required SDK versions). |
The SDK used to build MoltenVK is not necessarily the same as the one used at runtime. It can be newer than the host system. That’s why I assume
The way it works currently is the SDK version is specified by 1: Given that pkgsStatic and x86_64-darwin to aarch64-darwin cross both work now, it seems like only a matter of time before Linux to Darwin cross is also possible. I don’t even want to look at that before the stdenv is updated nor before some other housekeeping is done on the Darwin side (improving the SDK situation generally, etc). |
I had assumed we explicitly link it against our frameworks packaged in Nix but, looking at the outputs of Host platform wouldn't mean much for frameworks then; it'd be more of an intermediate between build and host platform.
Build and target/host platform might be different at some point, yes but I don't see why host and target platform would be different except for special cases like building a x86_64-darwin to aarch64-darwin compiler and the likes.
Doing that somehow doesn't feel right to me. Perhaps the stdenv could do some magic here; taking the correct SDK out of |
Yes. We link against stub frameworks, which point to the system frameworks but restrict the symbols to those available in that particular SDK version. Even the source-based SDKs still link against stubs for system frameworks other than for a handful that can be built from source.
That sounds more or less like the current situation with
The splicing or the attribute?
How would that look in a derivation? Would they continue to use |
|
To provide an example of what I mean, ideally the moltenvk = pkgs.darwin.apple_sdk_11_0.callPackage ../os-specific/darwin/moltenvk {
inherit (apple_sdk_11_0.frameworks) AppKit Foundation Metal QuartzCore;
inherit (apple_sdk_11_0) MacOSX-SDK Libsystem;
inherit (pkgs.darwin) cctools sigtool;
};To something like this: moltenvk = callPackage ../os-specific/darwin/moltenvk {
stdenv = overrideAppleSDK stdenv apple_sdk_latest;
};Inside the derivation, it would access the frameworks in some way. That pattern is what needs to be established. Is that |
|
I’m adding stdenvAdapters: add overrideSDK I ended up having to do something because curl needs to export the frameworks it links, and too many things (like Nix) link against The Packages that want to use the adapter should just use the unversioned SDK frameworks. Once there are more SDKs with newer frameworks not in the base SDK, we can start to figure out just how to handle that situation. In theory, if you use a newer SDK’s frameworks, it should also replace those with the requested one, but it might be confusing to use both the adapter and an SDK-specific framework attribute. I also wanted to thank everyone who participated in this discussion since it helped inform the approach taken in #263598. |
|
I mentioned this on Matrix, but I wanted to document it here. These are the known limitations of
|
This sounds like something we could fix, since the dylibs in the SDK are known? Or am I misunderstanding? |
It can be fixed. The issue is the SDKs expose different attributes under libs. Those need cleaned up and harmonized, but that’s not needed for staging-next, so it’s deferred until after the release and #229210. |
infinisil
added
the
significant
|
I’m working on a Darwin refactor that resolves the SDK situation. The solution ended up being that the SDK should be a normal package that can be added as an input. I have the SDKs building in this new pattern. I’m currently working on bootstrap changes and the transition away from frameworks to a single SDK for Darwin. The Darwin libc will just be a stub libc. The SDK has a hook that sets up You can see what this looks like for MoltenVK at https://github.com/reckenrode/nixpkgs/blob/darwin-sdk-refactor/pkgs/os-specific/darwin/moltenvk/default.nix. I also fixed xcbuild to pick up the SDK from the stdenv and work properly with structured attrs (so you can build configurations with spaces in their names). Ignore |
|
Closed by #346043 |
With #234710 getting close to done and the LLVM bump imminent, #229210 should be able to move forward adding new SDKs. We’ve had several conversations on Matrix regarding how we’d like to handle multiple SDKs. The purpose of this issue is to capture those conversations and work towards a consensus.
I’m pinging @toonn and @emilazy because we have discussed this on Matrix before. I am also pinging @jtojnar because @drupol mentioned having talked with him about the SDK issue. If there is anyone I forget, feel free to ping them.
Assumptions
Current Situation
There are two SDKs currently in nixpkgs: the 10.12 SDK and the 11.0 SDK. 10.12 is the default on x86_64-darwin, and 11.0 is the default on aarch64-darwin. The 10.12 SDK is built from Apple’s source releases where possible while the 11.0 SDK is derived from the upstream SDK. There is some shared implementation but also a lot of custom implementation between the two.
The default SDK is
darwin.apple_sdk. Packages that need to access SDK packages typically use the package and theinherit (darwin) <package>pattern. To use a different SDK, the common pattern is to useapple_sdk_<version>.callPackage, which overrides some parts of Darwin to provide versions from the new SDK. However, explicitly inheriting packages also requires updating those to reference the new SDK.Friction Points
The SDK typically ends up providing SDK-specific stdenvs and language frameworks (like
rustPlatform). The SDK should not have to concern itself with how it is being used downstream, and those frameworks should just worked based on the chosen SDK (picking it up as well as the updated compilers).xcbuild hardcodes the SDK it uses. It should be possible to override this in a way that does not require rebuilding xcbuild every time a new SDK is required.
propagatedBuildInputsdo not work well with SDK overrides. For example, it is not possible to mix libGLU, which propagates a couple of SDK frameworks, in a package that needs a different SDK because it results in headers being used from the wrong SDK.The top level of darwin attribute set contains some SDK-specific items (like Libsystem). These need to be moved to their appropriate SDKs. The top level should contain only packages that are Darwin-specific and part of or associated with an SDK.
CF is a massive pain in the ass. It should either be dropped or made to build on aarch64-darwin. Having two different build paths through the stdenv bootstrap complicates things. The way CF propagates also makes the stdenv bootstrap very sensitive to the order things are built on x86_64-darwin. There are also risks for cyclical dependencies¹.
Updating SDKs and making them source-based is difficult. This is going to be a problem for x86_64-darwin when it comes time to bump to clang 18 next year because libc++ will have dropped support for the platform, and additional post-10.12 APIs may be used (meaning it may not be feasible just to revert the commit that removed to 10.12 support).
¹: There are two examples I have encountered while working on the stdenv rework.
The text was updated successfully, but these errors were encountered: