Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdenvBootstrapTools: native aarch64-darwin build #202347

Merged
merged 2 commits into from
Mar 3, 2023
Merged

stdenvBootstrapTools: native aarch64-darwin build #202347

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
487e295
dumpnar: init at unstable-2023-01-01
stephank Jan 1, 2023
c3693fb
stdenvBootstrapTools: native aarch64-darwin build
stephank Jan 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
212 changes: 81 additions & 131 deletions pkgs/stdenv/darwin/make-bootstrap-tools.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,7 @@ in rec {
build = stdenv.mkDerivation {
name = "stdenv-bootstrap-tools";

nativeBuildInputs = [ buildPackages.nukeReferences buildPackages.cpio ]
++ lib.optionals targetPlatform.isAarch64 [ buildPackages.darwin.sigtool ];
nativeBuildInputs = [ nukeReferences dumpnar ];

buildCommand = ''
mkdir -p $out/bin $out/lib $out/lib/system $out/lib/darwin
Expand All @@ -49,20 +48,21 @@ in rec {

# Resolv is actually a link to another package, so let's copy it properly
cp -L ${lib.getLib darwin.Libsystem}/lib/libresolv.9.dylib $out/lib

cp -rL ${darwin.Libsystem}/include $out
chmod -R u+w $out/include
cp -rL ${darwin.ICU}/include* $out/include
cp -rL ${libiconv}/include/* $out/include
cp -rL ${gnugrep.pcre.dev}/include/* $out/include
mv $out/include $out/include-Libsystem
''}

cp -rL ${darwin.Libsystem}/include $out
chmod -R u+w $out/include
cp -rL ${darwin.ICU}/include* $out/include
cp -rL ${libiconv}/include/* $out/include
cp -rL ${lib.getDev gnugrep.pcre}/include/* $out/include
mv $out/include $out/include-Libsystem

# Copy coreutils, bash, etc.
cp ${coreutils_}/bin/* $out/bin
(cd $out/bin && rm vdir dir sha*sum pinky factor pathchk runcon shuf who whoami shred users)

cp ${bash}/bin/bash $out/bin
ln -s bash $out/bin/sh
Copy link
Contributor Author

@stephank stephank Jan 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bash and bzip2 links were previously created by the unpack script, but I moved them here. I feel like we should avoid altering the package at unpack-time if we can, but the remainder (wrappers, install names) seem unavoidable.

cp ${findutils}/bin/find $out/bin
cp ${findutils}/bin/xargs $out/bin
cp -d ${diffutils}/bin/* $out/bin
Expand All @@ -73,9 +73,11 @@ in rec {
cp ${gnutar}/bin/tar $out/bin
cp ${gzip}/bin/.gzip-wrapped $out/bin/gzip
cp ${bzip2_.bin}/bin/bzip2 $out/bin
ln -s bzip2 $out/bin/bunzip2
cp -d ${gnumake}/bin/* $out/bin
cp -d ${patch}/bin/* $out/bin
cp -d ${xz.bin}/bin/xz $out/bin
cp ${cpio}/bin/cpio $out/bin

# This used to be in-nixpkgs, but now is in the bundle
# because I can't be bothered to make it partially static
Expand Down Expand Up @@ -114,11 +116,9 @@ in rec {
cp -d ${lib.getLib pkgs.xar}/lib/libxar*.dylib $out/lib
cp -d ${pkgs.bzip2.out}/lib/libbz2*.dylib $out/lib

${lib.optionalString targetPlatform.isAarch64 ''
# copy sigtool
cp -d ${pkgs.darwin.sigtool}/bin/sigtool $out/bin
cp -d ${pkgs.darwin.sigtool}/bin/codesign $out/bin
''}
# copy sigtool
cp -d ${pkgs.darwin.sigtool}/bin/sigtool $out/bin
cp -d ${pkgs.darwin.sigtool}/bin/codesign $out/bin
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the conditions around codesigning to keep both archs more in sync. They don't harm intel, and only affect the bootstrap.


cp -d ${lib.getLib darwin.ICU}/lib/libicu*.dylib $out/lib
cp -d ${zlib.out}/lib/libz.* $out/lib
Expand All @@ -132,8 +132,9 @@ in rec {

cp -d ${lib.getLib darwin.libtapi}/lib/libtapi* $out/lib

${lib.optionalString targetPlatform.isx86_64 ''
cp -rd ${pkgs.darwin.CF}/Library $out
cp -rd ${pkgs.darwin.CF}/Library $out
${lib.optionalString stdenv.targetPlatform.isAarch64 ''
cp -rd ${pkgs.darwin.libobjc}/lib/* $out/lib/
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both CoreFoundation and libobjc are tbds only on aarch64, but having these allows the CF test to pass.

''}

chmod -R u+w $out
Expand All @@ -150,46 +151,44 @@ in rec {

# Strip executables even further
for i in $out/bin/*; do
if test -x $i -a ! -L $i; then
if [[ ! -L $i ]]; then
chmod +w $i
${stdenv.cc.targetPrefix}strip $i || true
fi
done

for i in $out/bin/* $out/lib/*.dylib $out/lib/darwin/*.dylib $out/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation; do
if test -x "$i" -a ! -L "$i"; then
echo "Adding rpath to $i"
for i in $out/bin/* $out/lib/*.dylib $out/lib/darwin/*.dylib; do
if [[ ! -L "$i" ]]; then
rpathify $i
fi
done

for i in $out/bin/*; do
if test -x "$i" -a ! -L "$i"; then
echo "Adding @executable_path to rpath in $i"
if [[ ! -L "$i" ]]; then
${stdenv.cc.targetPrefix}install_name_tool -add_rpath '@executable_path/../lib' $i
fi
done

${if stdenv.targetPlatform.isx86_64 then ''
rpathify $out/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
'' else ''
sed -i -e 's|/nix/store/.*/libobjc.A.dylib|@executable_path/../libobjc.A.dylib|g' \
$out/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation.tbd
Comment on lines +175 to +176
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a hack. I'm not sure how to properly implement the rpath logic here for tbds. But it seems to work fine, and only affects the bootstrap.

''}

nuke-refs $out/lib/*
nuke-refs $out/lib/system/*
nuke-refs $out/lib/darwin/*
nuke-refs $out/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
${lib.optionalString stdenv.targetPlatform.isx86_64 ''
nuke-refs $out/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
''}

mkdir $out/.pack
mv $out/* $out/.pack
mv $out/.pack $out/pack

mkdir $out/on-server
cp ${stdenv.shell} $out/on-server/sh
cp ${cpio}/bin/cpio $out/on-server
cp ${coreutils_}/bin/mkdir $out/on-server
cp ${bzip2_.bin}/bin/bzip2 $out/on-server

chmod u+w $out/on-server/*
${stdenv.cc.targetPrefix}strip $out/on-server/*
nuke-refs $out/on-server/*

(cd $out/pack && (find | cpio -o -H newc)) | bzip2 > $out/on-server/bootstrap-tools.cpio.bz2
dumpnar $out/pack | ${xz}/bin/xz > $out/on-server/bootstrap-tools.nar.xz
'';

allowedReferences = [];
Expand All @@ -204,98 +203,42 @@ in rec {

buildCommand = ''
mkdir -p $out/nix-support
echo "file tarball ${build}/on-server/bootstrap-tools.cpio.bz2" >> $out/nix-support/hydra-build-products
echo "file sh ${build}/on-server/sh" >> $out/nix-support/hydra-build-products
echo "file cpio ${build}/on-server/cpio" >> $out/nix-support/hydra-build-products
echo "file mkdir ${build}/on-server/mkdir" >> $out/nix-support/hydra-build-products
echo "file bzip2 ${build}/on-server/bzip2" >> $out/nix-support/hydra-build-products
echo "file tools ${build}/on-server/bootstrap-tools.nar.xz" >> $out/nix-support/hydra-build-products
'';
};

bootstrapLlvmVersion = llvmPackages.llvm.version;

bootstrapFiles = {
sh = "${build}/on-server/sh";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

nix-repl> :p legacyPackages.x86_64-darwin.freshBootstrapTools.test-pkgs.CoinMP
error: attribute 'sh' missing

       at /nix/store/8ycgcgzaskmw3wcw0kn7v0v1ggwnvbzz-source/pkgs/stdenv/darwin/default.nix:72:15:

           71|     name = "bootstrap-tools";
           72|     builder = bootstrapFiles.sh; # Not a filename! Attribute 'sh' on bootstrapFiles
             |               ^
           73|     args = if localSystem.isAarch64 then [ ./unpack-bootstrap-tools-aarch64.sh ] else [ ./unpack-bootstrap-tools.sh ];
«derivation

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverting this line show bzip2 is also required. So this PR might be incomplete.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: https://github.com/NixOS/nixpkgs/pull/202347/files#r1060701648

There's a tracking PR for new bootstrap tools: #222717

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok. I just track eval errors. I would have appreciated a throw or assert (in that case the :p can continue)

bzip2 = "${build}/on-server/bzip2";
mkdir = "${build}/on-server/mkdir";
cpio = "${build}/on-server/cpio";
tarball = "${build}/on-server/bootstrap-tools.cpio.bz2";
tools = "${build}/pack";
};

unpack = stdenv.mkDerivation (bootstrapFiles // {
name = "unpack";

# This is by necessity a near-duplicate of unpack-bootstrap-tools.sh. If we refer to it directly,
# we can't make any changes to it due to our testing stdenv depending on it. Think of this as the
# unpack-bootstrap-tools.sh for the next round of bootstrap tools.
# TODO: think through alternate designs, such as hosting this script as an output of the process.
buildCommand = ''
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved this to a file with -next suffix. The idea is that the regular file (without suffix) should always be identical when we finish a change, and it is much easier to diff the two.

# Unpack the bootstrap tools tarball.
echo Unpacking the bootstrap tools...
$mkdir $out
$bzip2 -d < $tarball | (cd $out && $cpio -i)

# Set the ELF interpreter / RPATH in the bootstrap binaries.
echo Patching the tools...
bootstrapTools = derivation {
inherit system;
Copy link
Member

@Artturin Artturin Sep 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks nix build -f pkgs/top-level/release.nix unstable

error: undefined variable 'system'
       at /home/artturin/nixgits/my-nixpkgs/.worktree/1/pkgs/stdenv/darwin/make-bootstrap-tools.nix:213:32:

          212|
          213|   bootstrapTools = derivation {
             |                                ^
          214|     inherit system;

it should probably be

diff --git a/pkgs/stdenv/darwin/make-bootstrap-tools.nix b/pkgs/stdenv/darwin/make-bootstrap-tools.nix
index 94c61e396b65..46ba25f8603b 100644
--- a/pkgs/stdenv/darwin/make-bootstrap-tools.nix
+++ b/pkgs/stdenv/darwin/make-bootstrap-tools.nix
@@ -211,7 +211,7 @@ in rec {
   };
 
   bootstrapTools = derivation {
-    inherit system;
+    inherit (localSystem) system;
 
     name = "bootstrap-tools";
     builder = "${bootstrapFiles.tools}/bin/bash";

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error can also be replicated in darwin repl with freshBootstrapTools.bootstrapTools

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#253759


export PATH=$out/bin

for i in $out/bin/*; do
if ! test -L $i; then
echo patching $i
install_name_tool -add_rpath $out/lib $i || true
fi
done
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step seems unnecessary, because we already add @executable_path/../lib during the build. Regardless, I based the new script on the ARM variant, which doesn't have it, and that appears to work nicely for Intel too.

name = "bootstrap-tools";
builder = "${bootstrapFiles.tools}/bin/bash";

ln -s libresolv.9.dylib $out/lib/libresolv.dylib

for i in $out/lib/*.dylib $out/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation; do
if test ! -L "$i" -a "$i" != "$out/lib/libSystem*.dylib"; then
echo "Patching $i"

id=$(otool -D "$i" | tail -n 1)
install_name_tool -id "$(dirname $i)/$(basename $id)" $i

libs=$(otool -L "$i" | tail -n +2 | grep -v libSystem | cat)
if [ -n "$libs" ]; then
install_name_tool -add_rpath $out/lib $i
fi
fi
done

ln -s bash $out/bin/sh
ln -s bzip2 $out/bin/bunzip2

# Provide a gunzip script.
cat > $out/bin/gunzip <<EOF
#!$out/bin/sh
exec $out/bin/gzip -d "\$@"
EOF
chmod +x $out/bin/gunzip

# Provide fgrep/egrep.
echo "#! $out/bin/sh" > $out/bin/egrep
echo "exec $out/bin/grep -E \"\$@\"" >> $out/bin/egrep
echo "#! $out/bin/sh" > $out/bin/fgrep
echo "exec $out/bin/grep -F \"\$@\"" >> $out/bin/fgrep
# This is by necessity a near-duplicate of patch-bootstrap-tools.sh. If we refer to it directly,
# we can't make any changes to it due to our testing stdenv depending on it. Think of this as the
# patch-bootstrap-tools.sh for the next round of bootstrap tools.
args = [ ./patch-bootstrap-tools-next.sh ];

cat >$out/bin/dsymutil << EOF
#!$out/bin/sh
EOF

chmod +x $out/bin/egrep $out/bin/fgrep $out/bin/dsymutil
'';
inherit (bootstrapFiles) tools;

allowedReferences = [ "out" ];
});
};

test = stdenv.mkDerivation {
name = "test";

realBuilder = "${unpack}/bin/bash";
realBuilder = "${bootstrapTools}/bin/bash";

tools = bootstrapTools;
buildCommand = ''
export PATH=${unpack}/bin
# Create a pure environment where we use just what's in the bootstrap tools.
export PATH=$tools/bin
wegank marked this conversation as resolved.
Show resolved Hide resolved

ls -l
mkdir $out
mkdir $out/bin
Expand All @@ -313,51 +256,58 @@ in rec {
# an SSL-capable curl
curl --version | grep SSL

${build}/on-server/sh -c 'echo Hello World'

# This approximates a bootstrap version of libSystem can that be
# assembled via fetchurl. Adapted from main libSystem expression.
mkdir libSystem-boot
cp -vr \
${darwin.darwin-stubs}/usr/lib/libSystem.B.tbd \
${darwin.darwin-stubs}/usr/lib/system \
${stdenv.cc.libc_dev}/lib/libSystem.B.tbd \
${stdenv.cc.libc_dev}/lib/system \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to work on both archs, but I think that's because the stubs are also already part of the libsystem build on the 10.12 SDK?

libSystem-boot

substituteInPlace libSystem-boot/libSystem.B.tbd \
--replace "/usr/lib/system/" "$PWD/libSystem-boot/system/"
ln -s libSystem.B.tbd libSystem-boot/libSystem.tbd
# End of bootstrap libSystem

export flags="-idirafter ${unpack}/include-Libsystem --sysroot=${unpack} -L${unpack}/lib -L$PWD/libSystem-boot"
export flags="-idirafter $tools/include-Libsystem --sysroot=$tools -L$tools/lib -L$PWD/libSystem-boot"

export CPP="clang -E $flags"
export CC="clang $flags -Wl,-rpath,${unpack}/lib -Wl,-v -Wl,-sdk_version,10.10"
export CXX="clang++ $flags --stdlib=libc++ -lc++abi -isystem${unpack}/include/c++/v1 -Wl,-rpath,${unpack}/lib -Wl,-v -Wl,-sdk_version,10.10"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -sdk_version appears unnecessary, and conflicts with -platform_version that the Clang driver already passes to ld. (It warns about -sdk_version being ignored.)

The -rpath is something Clang also accepts, no -Wl necessary.


echo '#include <stdio.h>' >> foo.c
echo '#include <float.h>' >> foo.c
echo '#include <limits.h>' >> foo.c
echo 'int main() { printf("Hello World\n"); return 0; }' >> foo.c
$CC -o $out/bin/foo foo.c
$out/bin/foo

echo '#include <CoreFoundation/CoreFoundation.h>' >> bar.c
echo 'int main() { CFShow(CFSTR("Hullo")); return 0; }' >> bar.c
$CC -F${unpack}/Library/Frameworks -framework CoreFoundation -o $out/bin/bar bar.c
$out/bin/bar

echo '#include <iostream>' >> bar.cc
echo 'int main() { std::cout << "Hello World\n"; }' >> bar.cc
$CXX -v -o $out/bin/bar bar.cc
$out/bin/bar
export CC="clang $flags -rpath $tools/lib"
export CXX="clang++ $flags --stdlib=libc++ -lc++abi -isystem$tools/include/c++/v1 -rpath $tools/lib"

# NOTE: These tests do a separate 'install' step (using cp), because
# having clang write directly to the final location apparently will make
# running the executable fail signature verification. (SIGKILL'd)
#
# Suspect this is creating a corrupt entry in the kernel cache, but it is
# unique to cctools ld. (The problem goes away with `-fuse-ld=lld`.)
Copy link
Contributor Author

@stephank stephank Jan 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is annoying and took long to debug. I've also seen it happen sporadically in Perl builds when it tries to run miniperl. Maybe we can fix cctools to write to a temporary file then rename. I've not yet looked at how LLD works and if it does something similar. I'm also suspicious Apple may have fixed this already, because Xcode's ld reports a much larger version number.


echo '#include <stdio.h>' >> hello1.c
echo '#include <float.h>' >> hello1.c
echo '#include <limits.h>' >> hello1.c
echo 'int main() { printf("Hello World\n"); return 0; }' >> hello1.c
$CC -o hello1 hello1.c
cp hello1 $out/bin/
$out/bin/hello1

echo '#include <CoreFoundation/CoreFoundation.h>' >> hello2.c
echo 'int main() { CFShow(CFSTR("Hullo")); return 0; }' >> hello2.c
$CC -F$tools/Library/Frameworks -framework CoreFoundation -o hello2 hello2.c
cp hello2 $out/bin/
$out/bin/hello2

echo '#include <iostream>' >> hello3.cc
echo 'int main() { std::cout << "Hello World\n"; }' >> hello3.cc
$CXX -v -o hello3 hello3.cc
cp hello3 $out/bin/
$out/bin/hello3

tar xvf ${hello.src}
cd hello-*
# stdenv bootstrap tools ship a broken libiconv.dylib https://github.com/NixOS/nixpkgs/issues/158331
am_cv_func_iconv=no ./configure --prefix=$out
make
make install

$out/bin/hello
'';
};
Expand Down
38 changes: 38 additions & 0 deletions pkgs/stdenv/darwin/patch-bootstrap-tools-next.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
set -euo pipefail

export PATH=$tools/bin

cp -R $tools $out
chmod -R u+w $out

updateInstallName() {
local path="$1"

cp "$path" "$path.new"
install_name_tool -id "$path" "$path.new"
codesign -f -i "$(basename "$path")" -s - "$path.new"
mv -f "$path.new" "$path"
}

find $out/lib -type f -name '*.dylib' -print0 | while IFS= read -r -d $'\0' lib; do
updateInstallName "$lib"
done

# Provide a gunzip script.
cat > $out/bin/gunzip <<EOF
#!$out/bin/sh
exec $out/bin/gzip -d "\$@"
EOF
chmod +x $out/bin/gunzip

# Provide fgrep/egrep.
echo "#! $out/bin/sh" > $out/bin/egrep
echo "exec $out/bin/grep -E \"\$@\"" >> $out/bin/egrep
echo "#! $out/bin/sh" > $out/bin/fgrep
echo "exec $out/bin/grep -F \"\$@\"" >> $out/bin/fgrep

cat >$out/bin/dsymutil << EOF
#!$out/bin/sh
EOF

chmod +x $out/bin/egrep $out/bin/fgrep $out/bin/dsymutil
Loading