Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debian packaging support #60690

Open
melizasw opened this issue Aug 22, 2022 · 112 comments
Open

Debian packaging support #60690

melizasw opened this issue Aug 22, 2022 · 112 comments
Labels
feature A feature request or implementation platform support

Comments

@melizasw
Copy link

melizasw commented Aug 22, 2022

@buttaface suggested that I start a new issue here to discuss problems that arise while I am packaging Swift for Debian. I have found the Fedora packaging that @tachoknight has done to be very helpful in this effort. A related work that I just learned of from @etcwilde is work on packaging for Ubuntu in the swift-installer-scripts repo.

The challenges are mainly getting Swift to build into a .deb package using the debuild tool and getting the packing into compliance with Debian policy. It has a very powerful linting tool that checks for many common errors and issues and most Debian Developers won't sign off on a package that isn't lint clean or reasonably close to it.

Here is the biggest issue that I know of right now. The file ConvertUTF.cpp uses a non-free license and is located in three places:

  1. indexstore-db/lib/LLVMSupport/Support/ConvertUTF.cpp
  2. llbuild/lib/llvm/Support/ConvertUTF.cpp
  3. llvm-project/llvm/lib/Support/ConvertUTF.cpp

Debian suggests replacing these files with libicu, which is already a part of Swift. But there are other alternatives out there as well such as utfcpp.
I am not a software engineer by trade so it is unlikely that I could make this replacement on my own without making a mess of things.

Another linting error that I can probably resolve post-bulid on my own is that many libraries have executable permissions. For example:
E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/10.0.0/lib/linux/libclang_rt.dyndd-x86_64.so]

And another example of how challenging this packaging can be, I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped. sigh

@finagolfin
Copy link
Member

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.

Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

@melizasw
Copy link
Author

melizasw commented Aug 22, 2022

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.

Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

That was a great idea. The llvm-toolchain-14 package has 840 lint errors including this one:
llvm-toolchain-14 | 1:14.0.6-2 |   | error | license-problem-convert-utf-code | [llvm/lib/Support/ConvertUTF.cpp]

I am able to see what they put in their copyright file and will do the same thing. The lint error will just have to be overridden. I see a lot of familiar lint errors in that llvm package so maybe this won't be as big of an issue as I had thought.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

I'm sure they do. I was just complaining how this turns into a three step process of disabling the stripping, disabling the lint errors that result, then probably having to explain to a Debian Developer why stripping was disabled.

Once I have the low hanging lint errors cleaned up I'll start working on getting the remaining errors reviewed by a Debian Developer to see if they have any concerns that I need to address.

@melizasw
Copy link
Author

melizasw commented Aug 23, 2022

Here is something else that would help me. Anything that lintian flags as an error is considered to be either a bug in the lint tool or a Debian policy violation. That said, it seems like there is some understanding that it isn't always that simple and overrides of rules are allowed.

For the following lintian errors it would be very helpful to know if the error is something I should be able to fix, or is expected. And if expected, what is the reason why. Any explanation that I can add to the comments of the lintian overrides will help to get approval to add the package to Debian. Most errors have multiple failures, but only the first is shown for brevity.

E: swift source: binary-with-bad-dynamic-table [llvm-project/llvm/test/tools/llvm-xray/X86/Inputs/elf64-objcopied-instrmap.bin]
N: 
N:   This appears to be an ELF file. According to readelf, the program headers
N:   suggests it should have a dynamic section, but readelf cannot find it.
N:   
N:   If it is meant to be external debugging symbols for another file, it
N:   should be installed under /usr/lib/debug. Otherwise, this could be a
N:   corrupt ELF file.
E: swiftlang: custom-library-search-path RUNPATH swift_5.6.2/build/buildbot_linux/llvm-linux-x86_64/lib [usr/libexec/swift/bin/lldb-argdumper]
N: 
N:   The binary or shared library sets RPATH or RUNPATH. This overrides the
N:   normal library search path, possibly interfering with local policy and
N:   causing problems for multilib, among other issues.
N:   
N:   The only time a binary or shared library in a Debian package should set
N:   RPATH or RUNPATH is if it is linked to private shared libraries in the
N:   same package. In that case, place those private shared libraries in
N:   /usr/lib/*package*. Libraries used by binaries in other packages should be
N:   placed in /lib or /usr/lib as appropriate, with a proper SONAME, in which
N:   case RPATH/RUNPATH is unnecessary.
N:   
N:   To fix this problem, look for link lines like: gcc test.o -o test
N:   -Wl,--rpath,/usr/local/lib or gcc test.o -o test -R/usr/local/lib and
N:   remove the -Wl,--rpath or -R argument. You can also use the chrpath
N:   utility to remove the RPATH.
N: 
N:   Please refer to https://wiki.debian.org/RpathIssue for details.

This should be pretty easy for me to fix if this is just an artifact of packaging and not a responsibility of the swift build scripts.

E: swift source: missing-notice-file-for-apache-license [swift-crypto/NOTICE.txt]
N: 
N:   The package appears to be licensed under the Apache 2.0 license and a
N:   NOTICE file (or similar) exists in the source tree. However, no files
N:   called NOTICE or NOTICE.txt are installed in any of the binary packages.
N:   
N:   The Apache 2.0 license requires distributing of such files:
N:   
N:    (d) If the Work includes a "NOTICE" text file as part of its
N:        distribution, then any Derivative Works that You distribute must
N:        include a readable copy of the attribution notices contained
N:        within such NOTICE file [..]
N:   
N:   Please include the file in your package, for example by adding
N:   path/to/NOTICE to a debian/package.docs file.
N: 
N:   Please refer to usr/share/common-licenses/Apache-2.0 for details.
E: swiftlang: no-code-sections [usr/libexec/swift/lib/swift_static/linux/libicudataswift.a]
N: 
N:   
N:   The named members of the static library have no usable code sections.
N:   
N:   It happens when shared objects are built with -flto=auto but without
N:   -ffat-lto-objects. dh_strip strips the LTO sections but may leave the
N:   static library without any usable code.
N: 
N:   Please refer to Bug#977596 for details.

These don't need to be executable do they? I'm not even sure just how they are getting +x.

E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/13.0.0/lib/linux/libclang_rt.asan-x86_64.so]
N: 
N:   Shared libraries should be mode 0644.
N: 
N:   Please refer to Run-time shared libraries (Section 8.1) in the Debian
N:   Policy Manual for details.

There are many of this kind. I'm not sure what the issue is as they don't go into the package. Maybe there is a way to clean them up just prior to running lintian?

E: swift source: source-is-missing [cmake/Tests/CMakeTests/ELF/elf32lsb.bin]
N: 
N:   The source of the following file is missing. Lintian checked a few
N:   possible paths to find the source, and did not find it.
N:   
N:   Please repack your package to include the source or add it to
N:   "debian/missing-sources" directory.
N:   
N:   Please note, that very-long-line-length-in-source-file tagged files are
N:   likely tagged source-is-missing. It is a feature not a bug.

This is just a warning, but it makes me wonder if I built Swift incorrectly? Maybe it doesn't matter as this is pertaining to the debug symbol package (swiftlang-dbgsym_5.6.2-1_amd64.deb) and I don't think that would be getting added.

W: swiftlang-dbgsym: debug-file-with-no-debug-symbols [usr/lib/debug/.build-id/00/bb2a059d939872ab355e8fa581567eac042f7d.debug]
N: 
N:   The binary is installed as a detached "debug symbols" ELF file, but it
N:   does not appear to have debug information associated with it.
N:   
N:   A common cause is not passing -g to GCC when compiling.
N:   
N:   Implementation detail: Lintian checks for the ".debug_line" and the
N:   ".debug_str" sections. If either of these are present, the binary is
N:   assumed to contain debug information.
N: 
N:   Please refer to Bug#668437 for details.

If anyone can help make sense of any of these lintian issues that would be most helpful.

@finagolfin
Copy link
Member

finagolfin commented Aug 23, 2022

  • The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.
  • The RUNPATH issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.
  • The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.
  • The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but willmay require you to bump the Swift package anytime the system libicu is updated.
  • Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.
  • I have no idea about that debug file with no symbols, haven't used that.

@melizasw
Copy link
Author

  • The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.

I looked up Debian's llvm-toolkit-14 package and wouldn't you know it, they have these same lintian errors and many of the others that I am seeing. If they can have them in llvm then swift should be allowed them as well. The lint tool looks at both the completed package as well as the source, so it seems to pick up all sorts of files that shouldn't matter.

* The `RUNPATH` issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.

I'm going to look into this some more, but maybe it isn't an issue. Based on the Debian wiki this forces the use of the custom Swift libraries, which I presume is better than picking up the system's libraries for llvm and others. Again, the Debian llvm package has these errors as well so I will contact the maintainer to find out what I can.

* The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.

For now I'm going to let the Debian build scripts copy over the file and see if that resolves things.

* The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but ~will~may require you to bump the Swift package anytime the system libicu is updated.

Well, wouldn't you know it. Debian's libicu-dev package has the exact same error on libicudata.a. So I guess nothing to worry about there.

* Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.here

I was able to get those fixed by manually running chmod at the appropriate place in the Debian packaging flow. There is a tool that is supposed to fix this automatically for me so maybe it has a bug. I will have to look into that. At no point in the build process do I see the swift tools setting these files to anything but 0644 and they don't become 0755 until they are copied out into the final directory structure so I'm not sure yet who is doing it.

* I have no idea about that debug file with no symbols, haven't used that.

Yah, probably not an issue.

I've just realized that the person that originally wanted to package Swift for Debian back in 2015 was and is a Debian Developer with the LLVM packaging team. If I can pique his interest to sponsor/mentor this package then he already has half of the answers and might even jump at the opportunity to get involved with it again. Crossing my fingers!

@melizasw
Copy link
Author

melizasw commented Aug 27, 2022

I'm at the point of trying to get a Debian Developer to sponsor my package and clean up any policy violations. The violation that was discovered first is that there is an existing Debian packaged named "python3-swiftclient" and they supply a binary named "swift". Since their binary does something completely different I am not allowed to have a binary named "swift" no matter how unlikely it is for someone to want to use the swift that is designed for server object storage and also compile Swift programs on that same server. I could use the "conflict" mechanism to prevent my package and theirs from being installed at the same time, but policy does not allow that unless the binaries provide the same functionality.

The recommended action is to contact the debian-devel mailing list to start a discussion of who has to change their name, or if both have to change their name. At the same time I could ask for an exception to the "conflict" mechanism rule.

I guess my main question here is that since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead? Would that break anything other than reams of documentation?

@finagolfin
Copy link
Member

since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead?

swift has two purposes: it invokes the REPL when run alone and it interprets a Swift file when given one. As such, you could always drop it as long as you don't care about those two uses, or look into renaming it instead.

Would that break anything other than reams of documentation?

No, I doubt there's even that much doc on it.

@melizasw
Copy link
Author

I don't know yet if this will be allowed by the Debian policy, but what I have done is create a set of scripts such that during the installation process if /usr/bin/swift does not exist the user is presented with a prompt asking if they wish to add it or not. If an actual binary file is located at /usr/bin/swift instead of a symbolic link then a message is printed out warning that "swift" already exists and it cannot be used to call the Swift REPL or compiler/interpreter.

@melizasw
Copy link
Author

I've been copying /usr/bin/ls to /usr/bin/swift to test the Debian package install in the case where the "swift" package is installed. If I forget to remove this copy before building Swift then it tries to build newswiftdriver with /usr/bin/swift and fails with a less than helpful message. I'm not sure what all I can do on my end to address this (and may not need to do anything), but I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

@finagolfin
Copy link
Member

finagolfin commented Aug 31, 2022

I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

@melizasw
Copy link
Author

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

I tried to reproduce this error on a Swift 5.5.1 build, but was unable to. Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case):

swift/utils/build-script --preset=buildbot_linux,no_test install_destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 installable_package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz
[swift/utils/build-script] NOTE: using preset "buildbot_linux,no_test", which expands to 

swift/utils/build-script --assertions --swift-enable-ast-verifier=0 --no-swift-stdlib-assertions '--swift-install-components=autolink-driver;compiler;clang-resource-dir-symlink;stdlib;swift-remote-mirror;sdk-overlay;parser-lib;toolchain-tools;license;sourcekit-inproc' '--llvm-install-components=llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld;LTO' --llbuild --swiftpm --swift-driver --xctest --libicu --libcxx --swiftdocc --build-ninja --install-llvm --install-swift --install-lldb --install-llbuild --install-swiftpm --install-swift-driver --install-xctest --install-libicu --install-prefix=/usr --install-libcxx --install-sourcekit-lsp --install-swiftdocc --build-swift-static-stdlib --build-swift-static-sdk-overlay --build-swift-stdlib-unittest-extra --test-installable-package --toolchain-benchmarks --install-destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 --installable-package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz --relocate-xdg-cache-home-under-build-subdir --build-subdir=buildbot_linux --lldb --release --test --validation-test --long-test --stress-test --test-optimized --foundation --libdispatch --indexstore-db --sourcekit-lsp --swiftdocc '--lit-args=-v --time-tests' --lldb-test-swift-only --install-foundation --install-libdispatch --reconfigure --skip-test-cmark --skip-test-lldb --skip-test-swift --skip-test-llbuild --skip-test-swiftpm --skip-test-swift-driver --skip-test-xctest --skip-test-foundation --skip-test-libdispatch --skip-test-playgroundsupport --skip-test-libicu --skip-test-indexstore-db --skip-test-sourcekit-lsp --skip-test-swiftdocc

[swift/utils/build-script] NOTE: Using toolchain default
+ cmake --version
+ mkdir -p /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux
+ rm -rf /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ cp -r /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/ninja /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ pushd /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ env CXX=/usr/bin/clang++ /usr/bin/python3 configure.py --bootstrap
[1/29][  3%][0.045s] INLINE build/browse_py.h
[2/29][  6%][0.057s] CXX build/debug_flags.o
[3/29][ 10%][0.635s] CXX build/browse.o
[4/29][ 13%][0.787s] CXX build/depfile_parser.o
[5/29][ 17%][0.971s] CXX build/disk_interface.o
[6/29][ 20%][1.180s] CXX build/clparser.o
[7/29][ 24%][1.208s] CXX build/edit_distance.o
[8/29][ 27%][1.449s] CXX build/dyndep.o
[9/29][ 31%][1.652s] CXX build/deps_log.o
[10/29][ 34%][2.033s] CXX build/clean.o
[11/29][ 37%][2.073s] CXX build/line_printer.o
[12/29][ 41%][2.137s] CXX build/dyndep_parser.o
[13/29][ 44%][2.163s] CXX build/build_log.o
[14/29][ 48%][2.252s] CXX build/lexer.o
[15/29][ 51%][2.430s] CXX build/graphviz.o
[16/29][ 55%][2.691s] CXX build/metrics.o
[17/29][ 58%][2.967s] CXX build/version.o
[18/29][ 62%][3.019s] CXX build/eval_env.o
[19/29][ 65%][3.108s] CXX build/string_piece_util.o
[20/29][ 68%][3.210s] CXX build/parser.o
[21/29][ 72%][3.246s] CXX build/util.o
[22/29][ 75%][3.353s] CXX build/graph.o
[23/29][ 79%][3.606s] CXX build/subprocess-posix.o
[24/29][ 82%][3.873s] CXX build/manifest_parser.o
[25/29][ 86%][3.964s] CXX build/build.o
[26/29][ 89%][4.120s] CXX build/state.o
[27/29][ 93%][4.192s] AR build/libninja.a
[28/29][ 96%][4.692s] CXX build/ninja.o
[29/29][100%][4.813s] LINK ninja
bootstrapping ninja...
warning: A compatible version of re2c (>= 0.11.3) was not found; changes to src/*.in.cc will not affect your build.
wrote build.ninja.
bootstrap complete.  rebuilding...
+ popd
--- Building earlyswiftdriver ---
Traceback (most recent call last):
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 722, in <module>
    sys.exit(main())
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 717, in main
    return main_normal()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 673, in main_normal
    invocation.execute()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 669, in execute
    self._execute(pipeline, all_host_names)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 727, in _execute
    self.execute_product_build_steps(product_class, host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 783, in execute_product_build_steps
    product.build(host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 69, in build
    run_build_script_helper('build', host_target, self, self.args)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 107, in run_build_script_helper
    swiftc_path = os.path.abspath(product.toolchain.swiftc)
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: command terminated with a non-zero exit status 1, aborting

I'm not likely to be doing any debug or pull requests on this any time soon as I'm focused on packaging.

@tachoknight
Copy link
Contributor

I had a similar issue with Swift being installed while packaging, but that was a mistake on my part (testing one version while building another) and the Fedora builders[1] won't have Swift installed by default.

[1] Dunno how Debian works, but all packages in the Fedora/RHEL environment are built from source on Fedora-controlled hardware, essentially in a special spun-up container; if I don't include it in the %BuildRequires section, it won't be in the container (which was how I discovered building Swift requires rsync 🙃).

@melizasw
Copy link
Author

Yes, it is the same with Debian. I should be using chroot and a clean build environment each time, but I'm not there yet. There is pretty much no chance that the official Debian package would ever get built with that other "swift" package installed. I have rsync in my build dependencies as well, not because I discovered that on my own, but from benefiting from your work. :-)

I've still got some things to sort out, but I'm going to ask if what I have now is good enough to upload to Debian as a starting point.

@tachoknight
Copy link
Contributor

out of curiosity, and because I don't have a Debian box handy, what is the other swift program? In Fedora's case there is another swift, but it's part of the OpenStack Object Storage API Python library, so the chance of a conflict with Swift® is effectively nil.

@melizasw
Copy link
Author

melizasw commented Aug 31, 2022

On Debian/Ubuntu/Pop_OS there is another package "swift" but it doesn't actually provide the binary /usr/bin/swift, that is provided by the related python-swiftclient and they are indeed part of the OpenStack Object Storage API Python library.

$ sudo nala install python3-swiftclient
$ which swift
/usr/bin/swift
$ swift --version
python-swiftclient 3.13.1

I'd prefer to have swiftlang refuse to install if python3-swiftclient is installed, but Debian policy prohibits two packages from providing the same binary unless they are are the same program having different implementations. I could ask for an exception to this policy, but I think prompting to make the link works too.

@tachoknight
Copy link
Contributor

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

@finagolfin
Copy link
Member

Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case)

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

@melizasw
Copy link
Author

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

@melizasw
Copy link
Author

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

Good to know. I've asked my package sponsor how best to handle this and mentioned this as being what Fedora has done. I think the only issue would be not having separate runtime libraries and needing to install swiftlang to get them. Have you done any work on exploring separating out the runtime libraries into their own package?

@finagolfin
Copy link
Member

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

Ah, you're right, found it. The current build checks for both swift and swiftc and tries to build the early swift driver if it finds just the former, as it assumes both will be installed together. Obviously, that assumption will break if your other OpenStack package is installed instead. I wouldn't worry about this, as very few will likely build the Swift toolchain, and almost nobody will do it with that other Swift package installed.

@melizasw
Copy link
Author

melizasw commented Sep 1, 2022

@tachoknight my Debian sponsor has asked me to move from the Swift provided clang, llvm, cmake, and icu packages to those that come with Debian. I know that lldb is customized by Swift as mentioned in your blog post, but what of these others? I'm assuming that icu and cmake would be fine as long as Swift doesn't depend on a specific version's features.

Do you know and/or do you have a blog post about that sort of issues that I can point him to? He is one of the Debian LLVM package maintainers so he is very experienced in this area.

Of course anyone else that knows is welcome to chime in too.

@tachoknight
Copy link
Contributor

This was my experience as well. The Swift-provided version of LLVM/LLDB is custom for Swift; the standard LLVM/LLDB you might install on any Debian or Fedora box knows nothing about Swift and it simply won't work. The solution I use in Fedora is to hide everything in /usr/libexec/swift and just symlink swift and swiftc to /usr/bin; the custom lldb executable for Swift is not on the path so there's no collision with an existing LLVM/LLDB installation.

Two things I can remember too: When I was originally getting it working with Fedora, I discovered that a lot of of the code assumes executables are in the same directory, or if dynamically loading a library, it will explicitly say something like ../lib/libFoo.so or whatever. The Swift toolchain is a complete package unto itself and thus it really needs to stay together; many early bugs with Swift on Fedora was simply due to constantly missing a spot where I had tried to redirect the loader to a more general location (e.g. Fedora wants all its 64-bit libs in /usr/lib64) and it was really frustrating. Thus when I was able to just drop everything into /usr/libexec/swift, it made life so much easier, and so many patches (and related bugs) just magically went away.
The other thing that has come up, and I want to say it's somewhere on the Swift forums, that there was some discussion about moving Swift upstream to be part of the LLVM project. I have no idea what, if any, plans there are to do that and there's no point in thinking about it (this is to head off the possibility of someone asking "Why don't they just move everything to the LLVM project?").

@melizasw
Copy link
Author

melizasw commented Sep 1, 2022

Thanks @tachoknight, I have forwarded your reply to my sponsor in the hopes that we will agree to keeping Swift together until upstream makes it possible for us to more easily split it apart.

@finagolfin
Copy link
Member

As for CMake and ICU, the Swift build just uses the upstream source for each of those, so no problem with not building those from source and using the prebuilt Debian packages instead, provided they're not really old.

The only issue is that if you use the system ICU, you may have to bump the Foundation build everytime the system ICU is updated, to make sure it's linked against the latest version, whereas you don't need to do that if you build another libicu for Swift and stash it separately with the Swift-forked lldb and so on.

@finagolfin
Copy link
Member

there was some discussion about moving Swift upstream to be part of the LLVM project.

Not Swift, I believe they just want to slowly upstream all Swift changes from their forked LLVM/clang/lldb to the upstream LLVM repo and get rid of those LLVM forks one day, but I'm guessing that's probably years away.

@tachoknight
Copy link
Contributor

Ah, that's right, I was remembering it incorrectly.

@melizasw
Copy link
Author

melizasw commented Sep 1, 2022

Thank you all for the information and help. My sponsor would like me to move to as many Debian packages as possible so here is our plan:

  1. Switch to Debian cmake and ninja (Debian ftpmasters will reject this package otherwise)
  2. Release swiftlang to Debian Experimental
  3. Switch to Debian icu, llvm, and clang
  4. Release to Debian Testing (from here it migrates to unstable after 5 days then to stable releases and derivative distributions)

@tachoknight
Copy link
Contributor

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

@melizasw
Copy link
Author

melizasw commented Sep 1, 2022

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

I'm hoping that will be open for negotiation. The desire is to not be packaging yet another fork of llvm/lldb/clang and having to deal with its bugs, but how do I make use of a tool chain that doesn't support my language? Start submitting all of Swift's changes as pull requests to Debian's LLVM packages? At that point it seems like you're still having to maintain a fork, but now those bugs impact far more Debian users than just the swiflang users. In any case, I must at least give it the old college try.

@melizasw
Copy link
Author

Swift 5.6.3 list the following valid hosts for Linux:

    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts. However, the only hosts from this list that Debian supports are x86_64, i686, aarch64, powerpc64, and s390x (powerpc and riscv64 as well). So I think I can just restrict my package to building on these 5 architectures.

x86_64 builds
i686 fails because it needs libatomic
aarch64 fails because its build is using the unsupported -fstack-clash protection
powerpc64 is also missing libatomic as well as whatever provides __gnu_f2h_ieee, __gnu_h2f_ieee, and __truncdfhf2
s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

I'm working to fix these, but any help would be appreciated.

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

@stevapple
Copy link
Contributor

Swift 5.6.3 list the following valid hosts for Linux:

    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts.

IIRC 32-bit host support was dropped long before (or hadn’t ever existed). Latest PRs that add new architectures show they’re adding support for targets that are meant to be cross-compiled against. 32-bit ARMv8 host triples should be handled similarly, while 64-bit ones can be mapped into aarch64.

s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

Maybe the port is not maintained any more and should be removed?

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

powerpc is target-only IIRC. riscv64 and other newly added architectures should be cross-compiled first. Not sure if the Debian packaging team will allow though. BTW this port doesn’t seem to be complete, so you can skip for now.

@stevapple
Copy link
Contributor

stevapple commented Oct 23, 2023

Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package.

@melizasw In fact, you can tweak the process to first build a toolchain without Swift in Swift support, and then build another one with it. This still works as for Swift 5.9, but will be removed at some time.

@melizasw
Copy link
Author

melizasw commented Oct 23, 2023

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Debian does indeed support the addition of new architectures and I was able to find some documentation on bootstrapping and cross-building which may be needed for riscv64 support. But for now I think I'm going to just focus on x86_64 and aarch64 and maybe s390x (Debian's LLVM toolchain builds for s390x).

Thanks for the help.

@tachoknight
Copy link
Contributor

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

@melizasw
Copy link
Author

melizasw commented Nov 4, 2023

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

Yep, I ran into the same problem and added to your bug report from 3+ years ago. There was some activity on the bug 2 months ago, but no fix yet. I guess no s390x for now.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

It'll take a while before I get there, but I will let you know.

On a RPI 4, it takes around 24 hours. 🙃

That beats the 2.5 days it took to build through QEMU! I ordered an RPI 5 that will be here in a few days to help out with that problem. I've also got a LicheePi 4A arriving later today to help me with building for riscv64 which I have been making some progress on by porting patches from @futurejones to 5.6.3.

@melizasw
Copy link
Author

melizasw commented Nov 25, 2023

Great news - Swift is now on Debian unstable (sid) and in a few days should go out to testing (trixie) on amd64 and arm64.
https://tracker.debian.org/pkg/swiftlang

Unfortunately when Ubuntu pulled it in and attempted to build on Noble it failed.
https://launchpad.net/ubuntu/+source/swiftlang

The issue appears to me to be related to the gold linker

[4402/4423][ 99%][4529.955s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4403/4423][ 99%][4529.991s] Linking CXX static library lib/libgtest.a
[4404/4423][ 99%][4530.020s] Linking CXX static library lib/libLLVMTestingSupport.a
[4405/4423][ 99%][4530.061s] Creating directories for 'compiler-rt'
[4406/4423][ 99%][4530.071s] No download step for 'compiler-rt'
[4407/4423][ 99%][4530.082s] No update step for 'compiler-rt'
[4408/4423][ 99%][4530.084s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
[4409/4423][ 99%][4530.094s] No patch step for 'compiler-rt'
[4409/4423][ 99%][4530.094s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - broken
CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL'
    
    Run Build Command(s): /usr/bin/ninja -v cmTC_d2d03
    [1/2][ 50%][0.017s] /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang   -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -MD -MT CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -c /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL/testCCompiler.c
    clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
    [2/2][100%][0.030s] : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    FAILED: cmTC_d2d03 
    : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    /usr/bin/ld: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
    clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:10 (project)


-- Configuring incomplete, errors occurred!
[4415/4423][ 99%][4530.818s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o
FAILED: tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure 
cd /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -DCMAKE_C_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_CXX_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ -DCMAKE_ASM_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DLLVM_CONFIG_PATH=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/llvm-config "-DLLVM_LIT_ARGS=-v --time-tests -j 4" -DCOMPILER_RT_OUTPUT_DIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./lib/clang/13.0.0 -DCOMPILER_RT_EXEC_OUTPUT_DIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin -DCOMPILER_RT_INSTALL_PATH:PATH=lib/clang/13.0.0 -DCOMPILER_RT_INCLUDE_TESTS=ON -DCMAKE_INSTALL_PREFIX=/usr -DLLVM_LIBDIR_SUFFIX= -DLLVM_RUNTIME_OUTPUT_INTDIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin -DCMAKE_OSX_DEPLOYMENT_TARGET= -DCMAKE_OSX_SYSROOT:PATH= -DCOMPILER_RT_INTERCEPT_LIBDISPATCH=ON -DCOMPILER_RT_PREFIX=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/projects/compiler-rt -DCOMPILER_RT_SRC_ROOT=/<<PKGBUILDDIR>>/llvm-project/llvm/../compiler-rt -GNinja -S /<<PKGBUILDDIR>>/llvm-project/llvm/../compiler-rt -B /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -E touch /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure
[4417/4423][ 99%][4531.167s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o
ninja: build stopped: subcommand failed.
-- Warning: {} Host toolchain could not locate a compiler to build swift-driver. (Try `--skip-early-swift-driver`)
--- Building cmark ---
Building the standard library for: swift-stdlib-linux-x86_64
Running Swift tests for: check-swift-only_early_swiftdriver-linux-x86_64 check-swift-all-linux-x86_64 check-swift-all-optimize-linux-x86_64
ERROR: command terminated with a non-zero exit status 1, aborting

The same build on Debian produces the following build log:

[4405/4423][ 99%][2053.044s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4406/4423][ 99%][2053.094s] Linking CXX static library lib/libgtest.a
[4407/4423][ 99%][2053.137s] Linking CXX static library lib/libgtest_main.a
[4408/4423][ 99%][2053.138s] Linking CXX static library lib/libLLVMTestingSupport.a
[4409/4423][ 99%][2053.157s] Creating directories for 'compiler-rt'
[4410/4423][ 99%][2053.175s] No download step for 'compiler-rt'
[4411/4423][ 99%][2053.193s] No update step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] No patch step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for unwind.h
-- Looking for unwind.h - found
-- Looking for rpc/xdr.h
-- Looking for rpc/xdr.h - not found
CMake Warning at cmake/Modules/CompilerRTUtils.cmake:352 (message):
  llvm-config finding testingsupport failed with status 1
Call Stack (most recent call first):
  CMakeLists.txt:81 (load_llvm_config)

I'm not sure why Debian is skipping the check but Ubuntu is not and then failing it. My Debian build doesn't have LLVMgold.so so no surprise that Ubuntu doesn't have it either.

I'm going to try the following to see if they help:

  1. Add binutils-dev to the build dependencies
  2. patch llvm-project-orig/llvm/cmake/config-ix.cmake as @tachoknight did for his Fedora 5.6.3 build to set LLVM_BINUTILS_INCDIR to /usr/include (Debian LLVM 16 also sets this path for its build).

Any help or advice would be appreciated as at the moment I am not set up to run test builds on Ubuntu and cannot directly test these without first releasing them to Debian then waiting for Ubuntu to pull it in and build.

@melizasw
Copy link
Author

melizasw commented Nov 30, 2023

More good news. Swift has now migrated to Debian testing and the two fixes mentioned above did indeed get the Ubuntu Noble build past that hang-up. I drug an old system out of retirement and installed Ubuntu on it so that I could work through the build errors on Noble.

Unfortunately now I'm getting another error, something about trying to link a shared library and needed to compile with -fPIC.

[770/863][ 89%][27.919s] Linking CXX shared library swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so
FAILED: swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so 

/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: warning: relocation against `__gwp_asan_default_options' in read-only section `.text._ZN8gwp_asan7options11initOptionsEPKcPFvS2_zE'
/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

That doesn't seem too hard, I just need to find the right place to do so when I get more time.

@futurejones
Copy link
Contributor

@melizasw I have installed and tried to test your debian packages but they don't work.
I am just trying to run a simple "hello world" using the swift package manager.
The errors are very easy to replicate.

The test environment is a Debian Trixie docker container.

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it debian:trixie /bin/bash
apt update
apt install swiftlang
mkdir hello && cd hello && swift package init --type executable && swift run
root@d5e164273bb5:/# mkdir hello && cd hello && swift package init --type executable && swift run
Creating executable package: hello
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/hello/main.swift
Creating Tests/
Creating Tests/helloTests/
Creating Tests/helloTests/helloTests.swift
'hello': error: invalidManifestFormat("<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"CoreFoundation.h\"\n        
 ^\n/usr/libexec/swift/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: \'sys/types.h\' file not found\n#include <sys/types.h>\n         
^\n/usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module \'CDispatch\' imported from /usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:\n#include <dispatch/dispatch.h>\n         
^\n<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"dispatch.h\"\n         ^\n/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:10: note: in file included from 
/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:\n#include <os/generic_unix_base.h>\n  
^\n/usr/libexec/swift/lib/swift/os/generic_unix_base.h:24:10: 
error: \'sys/param.h\' file not found\n#include <sys/param.h>\n         
^\n<unknown>:0: error: could not build C module \'CoreFoundation\'", diagnosticFile: nil)
root@d5e164273bb5:/hello# 

This error also occurs when using Debian Unstable.

When trying to build using swiftc we get an another error.

root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift 
error: link command failed with exit code 1 (use -v to see invocation)
clang-13: error: invalid linker name in argument '-fuse-ld=gold'

This indicates that binutils is missing from the required dependancies.
But after installing binutils we get another error.

root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift 
error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: error: cannot open Scrt1.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crti.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtbeginS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtendS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtn.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
/usr/bin/ld.gold: error: cannot find -lc
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

I have filed a bug with Debian to track the issue.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057117

@melizasw
Copy link
Author

@futurejones this is due to missing dependencies on binutils and gcc. If you install these two packages it will get you up and running for the next few days until I can get a new release out.

@futurejones
Copy link
Contributor

@melizasw there appears to be a problem with the relationship between the different swiftlang packages.

The swiftlang package is listing the swiftlang-dev package as a dependency, whereas the swiftlang-dev package only lists swiftlang as recommends`.

The swiftlang-dev package should list both swiftlang and libswiftlang as dependencies.
https://packages.debian.org/trixie/swiftlang-dev

The swiftlang package should not list swiftlang-dev as a dependency.
https://packages.debian.org/trixie/swiftlang

@melizasw
Copy link
Author

@futurejones I can certainly do that. Can you help me understand though what use swiftlang is without swiftlang-dev? I was under the impression that swiftlang wasn't of much use without both libswiflang and swiftland-dev.

@etcwilde
Copy link
Contributor

etcwilde commented Dec 18, 2023

So first, I'm really happy to see this work making progress. I think this will really help with the Swift-on-Linux story.

For some ideas on the separation of the libraries and tools, it might be worth looking at the Windows installer. I don't remember the exact split but it does a good job of separating concerns for developers, CI, and end-consumers of Swift programs. You shouldn't need to download the compiler if you just want to run a pre-compiled program that happens to be written in Swift, and you shouldn't need to download the headers for CoreFoundation if you just want to build a hello-world, for instance. It looks like that separation is mostly covered with the separation of swiftlang and libswiftlang, though perhaps could be split further for the stdlib and corelibs.

I may have missed a few runtime libraries in the listing here, so worth double-checking, but I think it's reasonable breakdown:

Runtimes:

  • swift runtimes (both with and without swiftmodules/header files (dev and non-dev versions))
    • swiftCore
    • swiftGlibc
    • swiftRemoteMirror
    • swiftSwiftOnoneSupport
    • swift_Concurrency
    • swift_Differentiation
    • swift_MatchingEngine
    • swift_StringProcessing
    • libdispatch*
  • corelibs
    • blocksRuntime
    • Foundation
    • FoundationNetworking
    • FoundationXML

* The challenge with libdispatch is that concurrency depends on it. The C bits of corelibs-libdispatch actually get built twice in the toolchain build, so I wonder if it would make sense to configure the build to emit a static archive so that we don't expose that. Then libdispatch.so could be installed as part of the corelibs package.

We should keep the runtime libraries together installed as one. I don't have a strong opinion on the corelibs libraries being together or apart, though mildly in favor of keeping them together as a swiftlang-corelibs package. Then we would want a dev variant of the swiftlang runtimes and swiftlang corelibs packages containing swiftmodules and headers. I think that keeping the corelibs separate from the runtimes is important, especially as work on swift-Foundation progresses, I'm not entirely sure what the future of corelibs-Foundation looks like.

Looking at swiftlang-dev, I'm concerned about the libcxx headers getting installed? If folks are doing C++ interop, which I don't think Swift 5.6 supports anyway, those headers will need to match the C++ runtime they're building for, which natively would be libstdc++ on the system.

Build Tools: (what I assume is the umbrella swiftlang package)

  • swiftc
  • swiftpm
  • clang

The swift-clang will be an interesting tricky bit. We'll want to provide the special swift-clang that has support for swift and swift-async calling conventions so that folks can use the C and C++ interop features. That of course conflicts with the existing clang package. I think the provides mechanism might work here though? A swift-clang package provides clang perhaps?

My concern with naming clang swift-clang is that various build-scripts may try to use clang while consuming a header emitted by Swift. Those headers may contain functions with swift or swiftasync calling conventions, so if they use the wrong C/C++ compiler, they will get a warning about the unknown attribute, but the program will compile with a misaligned ABI. On a good day, that will crash their program. On a bad day, their program will look like it runs, but has totally random garbage floating around. I'm open to push-back on that.

swiftlang can probably drop the headers and swiftmodules, and instead install those in the runtime-dev package. It's true that the compiler isn't terribly useful without them, but it's more conventional to install the headers associated with the library in the <library>-dev package. See the gcc vs libc6 vs libc6-dev filelists, for example. gcc doesn't come with headers, but libc6-dev comes with the C headers.

Dev Tools:

  • lldb
  • sourcekit-lsp
  • other tools and files useful for development but not strictly necessary for running or building a program

Same sort of situation with the swift-lldb as with clang. The lldb from apple/llvm-project has extra bits to make debugging Swift work, but that will conflict with the existing lldb package. Probably the same resolution since it provides lldb + swift extras.

Anyway, my initial thoughts. Thank you for working on this.

@melizasw
Copy link
Author

melizasw commented Aug 14, 2024

Despite what was said above about building Swift 5.7.3 using Swift 5.6.3, I tried without Swift and was able to build 5.7.3 just fine and am in the process of getting that pushed out into Debian. In the mean time I decided to build Swift 5.8.1 without using Swift 5.7.3, which I think should be possible as @tachoknight did so for Fedora. However, I got this bootstrapping error quite similar to what @stevapple got nearly 2 years ago in Windows - #61410

Here is my version of the error, or at least the start of about 5,000 lines of them.

[1215/1425][ 85%][1092.863s] Building swift module Basic
FAILED: bootstrapping1/SwiftCompilerSources/Basic.o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.o
cd /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources && /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping0/bin/swiftc -c -o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.o -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping0/bin/../lib -I /usr/lib -target x86_64-pc-linux-gnu -module-name Basic -emit-module -emit-module-path /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.swiftmodule -parse-as-library /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources/Sources/Basic/SourceLoc.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources/Sources/Basic/Utils.swift -wmo -Xfrontend -validate-tbd-against-ir=none -Xfrontend -enable-experimental-cxx-interop -Xcc -UIBOutlet -Xcc -UIBAction -Xcc -UIBInspectable -Xfrontend -disable-implicit-string-processing-module-import -O -cross-module-optimization -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-x86_64/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-x86_64/tools/clang/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/SwiftCompilerSources/../include -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources
:0: warning: unable to perform implicit import of "_Concurrency" module: no such module found
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include/swift/AST/ClangModuleLoader.h:19:10: note: while building module 'Clang_AST' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include/swift/AST/ClangModuleLoader.h:19:
#include "clang/AST/DeclTemplate.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/AST/APValue.h:16:10: note: while building module 'Clang_Basic' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/AST/APValue.h:16:
#include "clang/Basic/LLVM.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/Basic/Sanitizers.h:20:10: note: while building module 'LLVM_Transforms' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/Basic/Sanitizers.h:20:
#include "llvm/Transforms/Instrumentation/AddressSanitizerOptions.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h:20:10: note: while building module 'LLVM_intrinsic_gen' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h:20:
#include "llvm/IR/PassManager.h"
^
:1:10: note: in file included from :1:
#include "IR/Argument.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Argument.h:17:10: note: in file included from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Argument.h:17:
#include "llvm/IR/Attributes.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Attributes.h:30:10: note: in file included from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Attributes.h:30:
#include
^
/usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/set:62:10: note: in file included from /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/set:62:
#include <bits/stl_tree.h>
^
/usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_tree.h:2005:5: error: missing '#include "gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_pair.h"'; 'pair' must be declared before it is used
pair<typename _Rb_tree<_Key, _Val, _KeyOfValue,
^
/usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_pair.h:284:12: note: declaration here is not visible
struct pair
^

I can't figure out what is causing this error. Is there a way around it or do I need to give up and build 5.8.1 with 5.7.3?

@melizasw
Copy link
Author

I've not been able to make any progress on the above error when trying to build Swift 5.8.1 without a Swift compiler installed. I tried turning off bootstrapping, but that gave me errors that were equally as confusing. I am mostly trying to build without a Swift compiler as I still have hope that I can get a riscv build going and support for it looks to be better in 5.8.1 than in 5.7.3 and I still don't have it working yet.

Another reason that I was not building with a swift compiler installed is that that flow was broken too so I figured I may as well try one way then the other. Building with a swift compiler was broken because on an x86_64 Debian system clang will identify as pc-linux-gnu so I had to build swift 5.7.3 for pc-linux-gnu and swift/utils/swift_build_support/swift_build_support/products/product.py is hard coded to use unknown-linux. This was causing the build scripts to not detected a working swift compiler on x86_64. Arm64 and RiscV Debian systems identify as unknown-linux-gnu and thus are not affected. It's not a Swift bug per se as I had to edit swift/cmake/modules/SwiftConfigureSDK.cmake to build as pc-linux-gnu.

Now that I'm trying to build Swift 5.8.1 using an installed swift 5.7.3, do I need to set bootstrapping to hosttools?

@melizasw
Copy link
Author

Problem resolved, though I certainly took the scenic route to get there. The issue is covered in good detail in #65028 so no need to repeat it. Swift 5.8.x apparently introduced some cmake file changes that are not compatible with cmake 3.25 and newer. Once I uninstalled Debian's cmake the build reverted to the specified version of cmake (3.19.6) and this build error went away. Looks like the patch will be in 5.9.x.

@melizasw
Copy link
Author

melizasw commented Nov 1, 2024

I was doing well building 5.8.1 when Debian switched to Clang 19 which introduced -Wcast-function-type-mismatch and then enabled it by default. This has caused an error building swift-corelibs-libdispatch which I have reported on github in case they wish to fix it. For now I'd just like to add -Wno-cast-function-type-mismatch to the flags for swift-corelibs-libdispatch to turn off the warning/error. Can someone point me in the right direction to inject this flag into the swift build scripts?

@yuzibo
Copy link

yuzibo commented Nov 4, 2024

@melizasw Many thanks for your work on Debian packaging.
I noticed you have uploaded 5.8.1 to Debian also, that is great. I am a Debian riscv porter, so I am looking into how to build it on riscv64 now on Debian.

Now the issue is below:

...
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang - broken
CMake Error at /usr/share/cmake-3.30/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/Tr
yCompile-BalgBc'

    Run Build Command(s): /usr/bin/ninja -v cmTC_587eb
   [1/2][ 50%][0.460s] /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang   -g -O2 -Werror=implicit-function-declaration
 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -MD -MT CMakeFiles/cmTC_587eb.dir/testCCo
mpiler.c.o -MF CMakeFiles/cmTC_587eb.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_587eb.dir/testCCompiler.c.o -c /<<PKGBUILDDIR>>/build/bui
ldbot_linux/llvm-linux-riscv64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-BalgBc/testCCompiler.c
    [2/2][100%][0.888s] : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang -g -O2 -Werror=implicit-function-declarat
ion -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro CMakeFiles/cmTC_587eb.dir
/testCCompiler.c.o -o cmTC_587eb   && :
    FAILED: cmTC_587eb
    : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang -g -O2 -Werror=implicit-function-declaration -ffile-prefix-ma
p=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro CMakeFiles/cmTC_587eb.dir/testCCompiler.c.o -
o cmTC_587eb   && :
    /usr/bin/ld: cannot find Scrt1.o: No such file or directory
    clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.
...

although the log is suggestive enough, but I am not convinced a little. Could you have an idea about this? Certainly, I will continue to try it on my side. TIA.

@melizasw
Copy link
Author

melizasw commented Nov 4, 2024

@yuzibo Yes, I got 5.8.1 into Debian by using the clang-18 package as my workaround for now. However, the arm64 build passes with Clang 16 and fails with Clang 18 so I have my work cut out for me there.

Yes, I am familiar with the ld cannot find Scrt1.o error on RISCV. I've been slowly chipping away at it, but the patches in Debian are close, but not complete yet. My build is still running so I don't know yet if these are enough, but try adding them to your patches:

--- swiftlang-5.8.1.orig/swift/utils/build-presets.ini
+++ swiftlang-5.8.1/swift/utils/build-presets.ini
@@ -800,7 +800,9 @@ no-swift-stdlib-assertions
 [preset: mixin_linux_install_components_with_clang]
 
 swift-install-components=autolink-driver;compiler;clang-resource-dir-symlink;stdlib;swift-remote-mirror;sdk-overlay;static-mirror-lib;toolchain-tools;license;sourcekit-inproc
-llvm-install-components=llvm-ar;llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld;LTO;clang-features-file
+llvm-install-components=llvm-ar;llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;clangd;lld;LTO;clang-features-file
+
+skip-build-compiler-rt
 
 [preset: mixin_linux_installation]
 mixin-preset=


@@ -1475,7 +1475,7 @@ mixin-preset=
 [preset: LLDB_Nested]
 skip-build-benchmarks
 install-destdir=%(swift_install_destdir)s
-llvm-targets-to-build=X86;ARM;AArch64;PowerPC;SystemZ;Mips
+llvm-targets-to-build=X86;ARM;AArch64;PowerPC;SystemZ;Mips;RISCV
 
 [preset: LLDB_Swift_DebugAssert]
 mixin-preset=

If these work then I'll need to figure out a clean way to only enable them for RISCV and not the other architectures.

I've gathered my RISCV patches from various sources, but these two changes are thanks to @futurejones and his patches for 5.8 - https://github.com/swift-riscv/swift-riscv64/tree/bcb3fad816f5dd89fcc38804a5d5a1b0a58a1bc3/patches/release-5.8-branch I've not incorporated all of his patches as I try to only enable them as needed to fix a specific error so that I can understand the need for each one and make sure I don't grab any that are not needed for Debian.

I've already got 5.9.2 building on x86_64 with a build depends on 5.8.1 so RISCV really needs to get onboard at 5.8.1 or it will get left behind and require some cross building or bootstrapping later on.

@yuzibo
Copy link

yuzibo commented Nov 4, 2024

@melizasw Many thanks for your sharing about this, I will try it on my side.

BTW, do you need one real riscv64 hardware to debug something on here? if so, I can offer you with access remote with ssh.

@yuzibo
Copy link

yuzibo commented Nov 4, 2024

@melizasw It moved forward a lot with your patch. now it failed with the below:

public/CMakeFiles/swiftDemangling-linux-riscv64.dir/__/__/lib/Demangling/Errors.cpp.o.d -o stdlib/public/CMakeFiles/swiftDemangling-linux
-riscv64.dir/__/__/lib/Demangling/Errors.cpp.o -c /<<PKGBUILDDIR>>/swift/lib/Demangling/Errors.cpp
In file included from /<<PKGBUILDDIR>>/swift/lib/Demangling/Errors.cpp:19:
In file included from /<<PKGBUILDDIR>>/swift/include/swift/Demangling/Errors.h:24:
In file included from /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/lib/clang/13.0.0/include/inttypes.h:21:
In file included from /usr/include/inttypes.h:25:
In file included from /usr/include/features.h:402:
/usr/include/features-time64.h:20:10: fatal error: 'bits/wordsize.h' file not found
#include <bits/wordsize.h>
         ^~~~~~~~~~~~~~~~~
1 error generated.
[48/1463][  3%][2.057s] Copying Target.h to /<<PKGBUILDDIR>>/build/buildbot_linux/swift-linux-riscv64/./lib/swift/shims
[49/1463][  3%][2.058s] Copying UnicodeData.h to /<<PKGBUILDDIR>>/build/buildbot_linux/swift-linux-riscv64/./lib/swift/shims
[50/1463][  3%][2.059s] Copying ThreadLocalStorage.h to /<<PKGBUILDDIR>>/build/buildbot_linux/swift-linux-riscv64/./lib/swift/shims
[51/1463][  3%][2.170s] Copying module.modulemap to /<<PKGBUILDDIR>>/build/buildbot_linux/swift-linux-riscv64/./lib/swift/shims
...

It seems it has finished the first stage of building. I am looking at link you gave me above.

@melizasw
Copy link
Author

melizasw commented Nov 4, 2024

Yes, I am getting the same error now. It can't seem to locate /usr/include/riscv64-linux-gnu/bits/wordsize.h even though it has no problem doing so on x86_64 and arm64. I will try to dig into this tonight after work. This is the same error that I got on 5.7.3 that stopped me and I did not get past. My best guess is an issue configuring include paths in the CMake files.

@tachoknight
Copy link
Contributor

tachoknight commented Nov 5, 2024

I was doing well building 5.8.1 when Debian switched to Clang 19 which introduced -Wcast-function-type-mismatch and then enabled it by default. This has caused an error building swift-corelibs-libdispatch which I have reported on github in case they wish to fix it. For now I'd just like to add -Wno-cast-function-type-mismatch to the flags for swift-corelibs-libdispatch to turn off the warning/error. Can someone point me in the right direction to inject this flag into the swift build scripts?

I added this to my build script and it works now:

export CFLAGS="-Wno-error=cast-function-type `echo $CFLAGS`"

@tachoknight
Copy link
Contributor

@melizasw It turned out to be a little bit more difficult than what I initially wrote. Here's the patch file I addd to make it work reliably. Hopefully that helps. :)

@melizasw
Copy link
Author

melizasw commented Nov 17, 2024

@melizasw It turned out to be a little bit more difficult than what I initially wrote. Here's the patch file I addd to make it work reliably. Hopefully that helps. :)

Yah, I ended up using this patch:

--- swiftlang-5.8.1.orig/swift-corelibs-libdispatch/cmake/modules/DispatchCompilerWarnings.cmake
+++ swiftlang-5.8.1/swift-corelibs-libdispatch/cmake/modules/DispatchCompilerWarnings.cmake
@@ -65,6 +65,7 @@ else()
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-used-but-marked-unused>)
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-void-pointer-to-int-cast>)
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-vla>)
+  add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-error=cast-function-type-mismatch>)
 
   if(CMAKE_SYSTEM_NAME STREQUAL Android)
     add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-incompatible-function-pointer-types>)

The issue now is that 5.8.1 builds fine on amd64, but fail with a stack dump on arm64 with Clang 18 and 19. I'm a little stumped, but haven't spent a lot of time on it yet. Might be related to #70345

[1418/1463][ 96%][5364.842s] Compiling /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest//LINUX/aarch64/StdlibUnittest.o
FAILED: stdlib/private/StdlibUnittest/LINUX/aarch64/StdlibUnittest.o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest/LINUX/aarch64/StdlibUnittest.o 
cd /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest && /usr/bin/cmake -E env LD_LIBRARY_PATH=/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bootstrapping1/lib/swift/linux /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/binforpython/python /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/utils/line-directive @/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest/20a38dcb7fd74f7c606e850bcb913edec34a43c2.txt -- /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./bin/swiftc -c -sdk / -target aarch64-unknown-linux-gnu -resource-dir /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift -O -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING -D SWIFT_RUNTIME_OS_VERSIONING -D SWIFT_STDLIB_ENABLE_UNICODE_DATA -D SWIFT_STDLIB_ENABLE_VECTOR_TYPES -D SWIFT_STDLIB_HAS_COMMANDLINE -D SWIFT_STDLIB_HAS_STDIN -D SWIFT_STDLIB_HAS_ENVIRON -Xcc -DSWIFT_STDLIB_HAS_ENVIRON -D SWIFT_THREADING_LINUX -tools-directory /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-aarch64/./bin -module-cache-path /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./module-cache -no-link-objc-runtime -enable-library-evolution -library-level api -Xfrontend -require-explicit-availability=ignore -Xfrontend -enforce-exclusivity=unchecked -D SWIFT_ENABLE_REFLECTION -module-name StdlibUnittest -swift-version 5 -runtime-compatibility-version none -disable-autolinking-runtime-compatibility-dynamic-replacements -Xfrontend -disable-autolinking-runtime-compatibility-concurrency -Xfrontend -disable-objc-interop -Xfrontend -disable-objc-attr-requires-foundation-module -DSWIFT_ENABLE_REFLECTION -warn-implicit-overrides -Xfrontend -enable-ossa-modules -Xfrontend -enable-lexical-lifetimes=false -Xfrontend -disable-implicit-concurrency-module-import -Xfrontend -disable-implicit-string-processing-module-import -Xfrontend -disable-implicit-string-processing-module-import -Xfrontend -prespecialize-generic-metadata -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 9999:macOS\ 9999,\ iOS\ 9999,\ watchOS\ 9999,\ tvOS\ 9999 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.0:macOS\ 10.14.4,\ iOS\ 12.2,\ watchOS\ 5.2,\ tvOS\ 12.2 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.1:macOS\ 10.15,\ iOS\ 13.0,\ watchOS\ 6.0,\ tvOS\ 13.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.2:macOS\ 10.15.4,\ iOS\ 13.4,\ watchOS\ 6.2,\ tvOS\ 13.4 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.3:macOS\ 11.0,\ iOS\ 14.0,\ watchOS\ 7.0,\ tvOS\ 14.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.4:macOS\ 11.3,\ iOS\ 14.5,\ watchOS\ 7.4,\ tvOS\ 14.5 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.5:macOS\ 12.0,\ iOS\ 15.0,\ watchOS\ 8.0,\ tvOS\ 15.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.6:macOS\ 12.3,\ iOS\ 15.4,\ watchOS\ 8.5,\ tvOS\ 15.4 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.7:macOS\ 13.0,\ iOS\ 16.0,\ watchOS\ 9.0,\ tvOS\ 16.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.8:macOS\ 13.3,\ iOS\ 16.4,\ watchOS\ 9.4,\ tvOS\ 16.4 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.9:macOS\ 9999,\ iOS\ 9999,\ watchOS\ 9999,\ tvOS\ 9999 -Xfrontend -target-min-inlining-version -Xfrontend min -module-link-name swiftStdlibUnittest -whole-module-optimization -parse-as-library -resource-dir /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift/linux -o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest//LINUX/aarch64/StdlibUnittest.o @/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest/20a38dcb7fd74f7c606e850bcb913edec34a43c2.txt
swift-frontend: /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/ADT/DenseMap.h:410: void llvm::DenseMapBase<llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function *>, swift::irgen::LinkEntity, llvm::Function *, llvm::DenseMapInfo<swift::irgen::LinkEntity>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function *>>::moveFromOldBuckets(BucketT *, BucketT *) [DerivedT = llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function *>, KeyT = swift::irgen::LinkEntity, ValueT = llvm::Function *, KeyInfoT = llvm::DenseMapInfo<swift::irgen::LinkEntity>, BucketT = llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function *>]: Assertion `!FoundVal && "Key already in new map?"' failed.
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
Stack dump:
0.	Program arguments: /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend -frontend -c /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/StdlibUnittest.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/CheckStrideable.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/InspectValue.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/LifetimeTracked.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/MinimalTypes.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/OpaqueIdentityFunctions.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/RaceTest.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/Statistics.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/StdlibCoreExtras.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/StringConvertible.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/StringTestHelpers.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/TestHelpers.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/stdlib/private/StdlibUnittest/TypeIndexed.swift -supplementary-output-file-map /tmp/supplementaryOutputs-1fc52d -target aarch64-unknown-linux-gnu -Xllvm -aarch64-use-tbi -disable-objc-interop -sdk / -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift/linux -warn-implicit-overrides -enable-library-evolution -module-cache-path /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./module-cache -module-link-name swiftStdlibUnittest -resource-dir /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift -swift-version 5 -tools-directory /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-aarch64/./bin -O -library-level api -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING -D SWIFT_RUNTIME_OS_VERSIONING -D SWIFT_STDLIB_ENABLE_UNICODE_DATA -D SWIFT_STDLIB_ENABLE_VECTOR_TYPES -D SWIFT_STDLIB_HAS_COMMANDLINE -D SWIFT_STDLIB_HAS_STDIN -D SWIFT_STDLIB_HAS_ENVIRON -D SWIFT_THREADING_LINUX -D SWIFT_ENABLE_REFLECTION -D SWIFT_ENABLE_REFLECTION -require-explicit-availability=ignore -enforce-exclusivity=unchecked -disable-autolinking-runtime-compatibility-concurrency -disable-objc-interop -disable-objc-attr-requires-foundation-module -enable-ossa-modules -enable-lexical-lifetimes=false -disable-implicit-concurrency-module-import -disable-implicit-string-processing-module-import -disable-implicit-string-processing-module-import -prespecialize-generic-metadata -define-availability "SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -define-availability "SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2" -define-availability "SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0" -define-availability "SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4" -define-availability "SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0" -define-availability "SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5" -define-availability "SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0" -define-availability "SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4" -define-availability "SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0" -define-availability "SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4" -define-availability "SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -target-min-inlining-version min -Xcc -DSWIFT_STDLIB_HAS_ENVIRON -parse-as-library -module-name StdlibUnittest -o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest//LINUX/aarch64/StdlibUnittest.o -runtime-compatibility-version none -disable-autolinking-runtime-compatibility-dynamic-replacements
1.	Swift version 5.8.1 (swift-5.8.1-RELEASE)
2.	Compiling with the current language version
3.	While evaluating request IRGenRequest(IR Generation for module StdlibUnittest)
4.	While emitting prespecialized metadata for type 'Optional<Int>'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x6570028) [0x5555f8b10028]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x656e06c) [0x5555f8b0e06c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x6570450) [0x5555f8b10450]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0x7fff002107b0]
/lib/aarch64-linux-gnu/libc.so.6(+0x87240) [0x7ffeff627240]
/lib/aarch64-linux-gnu/libc.so.6(gsignal+0x20) [0x7ffeff5d67a0]
/lib/aarch64-linux-gnu/libc.so.6(abort+0xf8) [0x7ffeff5c1a48]
/lib/aarch64-linux-gnu/libc.so.6(+0x2f860) [0x7ffeff5cf860]
/lib/aarch64-linux-gnu/libc.so.6(+0x2f8d4) [0x7ffeff5cf8d4]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1190f38) [0x5555f3730f38]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1190dc8) [0x5555f3730dc8]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1190d14) [0x5555f3730d14]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x11888dc) [0x5555f37288dc]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x126a590) [0x5555f380a590]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1269d68) [0x5555f3809d68]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x12699ac) [0x5555f38099ac]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x11f9628) [0x5555f3799628]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1178f64) [0x5555f3718f64]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1063738) [0x5555f3603738]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x10af384) [0x5555f364f384]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x107309c) [0x5555f361309c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x106463c) [0x5555f360463c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd6ac54) [0x5555f330ac54]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd67028) [0x5555f3307028]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd66190) [0x5555f3306190]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd76f70) [0x5555f3316f70]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd687e0) [0x5555f33087e0]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd679f0) [0x5555f33079f0]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xbdad64) [0x5555f317ad64]
/lib/aarch64-linux-gnu/libc.so.6(+0x2229c) [0x7ffeff5c229c]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x9c) [0x7ffeff5c237c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xbd9df0) [0x5555f3179df0]
<unknown>:0: error: unable to execute command: Aborted
<unknown>:0: error: compile command failed due to signal 6 (use -v to see invocation)
[1419/1463][ 96%][5365.217s] Compiling /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing//LINUX/aarch64/_StringProcessing.o
FAILED: stdlib/public/StringProcessing/LINUX/aarch64/_StringProcessing.o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing/LINUX/aarch64/_StringProcessing.o 

@melizasw
Copy link
Author

Confirmed that 5.8.1 build failures on Debian Sid with default Clang package (Clang 19) on Arm64 are fixed with the patch provided with bug report #70345

Assertion `!FoundVal && "Key already in new map?"' failed.

@melizasw
Copy link
Author

melizasw commented Nov 18, 2024

I have 5.9.2 building fine on x86_64, but on aarch64:

clang: error: no such file or directory: '/usr/libexec/swift/lib/swift/linux/x86_64/swiftrt.o'

Sigh

ETA: Self-induced with a patch that hard coded that path.

@melizasw
Copy link
Author

melizasw commented Nov 24, 2024

On to the next issue with packaging 5.9.2. It builds on x86_64 but not aarch64. I've traced it down to an error parsing json by swiftpm/Utilities/bootstrapping because switfc outputs some non-json on aarch64 that trips up the json parser.

x86_64:

usr/bin/swiftc -print-target-info
{
  "compilerVersion": "Swift version 5.9.2 (swift-5.9.2-RELEASE)",
  "target": {
    "triple": "x86_64-pc-linux-gnu",
    "unversionedTriple": "x86_64-pc-linux-gnu",
    "moduleTriple": "x86_64-pc-linux-gnu",
    "compatibilityLibraries": [ ],
    "librariesRequireRPath": false
  },
  "paths": {
    "runtimeLibraryPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux"
    ],
    "runtimeLibraryImportPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux",
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux/x86_64"
    ],
    "runtimeResourcePath": "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift"

aarch64:

usr/bin/swiftc -print-target-info
swift runtime: unable to protect path to swift-backtrace at 0x7fff4c7e5000: 22; disabling backtracing.
swift runtime: unable to protect environment for swift-backtrace at 0x7fff4c7dd000: 22; disabling backtracing.
swift runtime: unable to protect path to swift-backtrace at 0x7fff56635000: 22; disabling backtracing.
swift runtime: unable to protect environment for swift-backtrace at 0x7fff5662d000: 22; disabling backtracing.
{
  "compilerVersion": "Swift version 5.9.2 (swift-5.9.2-RELEASE)",
  "target": {
    "triple": "aarch64-unknown-linux-gnu",
    "unversionedTriple": "aarch64-unknown-linux-gnu",
    "moduleTriple": "aarch64-unknown-linux-gnu",
    "compatibilityLibraries": [ ],
    "librariesRequireRPath": false
  },
  "paths": {
    "runtimeLibraryPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux"
    ],
    "runtimeLibraryImportPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux",
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux/aarch64"
    ],
    "runtimeResourcePath": "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift"
  }
}

Swiftc 5.8.1 on aarch64 does not print this message. Does anyone have an idea on this one?
Maybe this? https://forums.swift.org/t/steps-to-resolve-unable-to-protect-swift-backtrace-disabling-backtracing/72076/2

@melizasw
Copy link
Author

5.9.2 is now building on Arm after modifying swift/include/swift/Runtime/Config.h to match Debian's page sizes.

 /// What is the system page size?
-#if defined(__APPLE__) && defined(__arm64__)
+#if defined(__aarch64__)
   // Apple Silicon systems use a 16KB page size
   #define SWIFT_PAGE_SIZE 16384
 #else

@melizasw
Copy link
Author

Just a happy little update.

With version 5.8.1 I was finally able to figure out why my builds always failed on Ubuntu. It turns out to have been a "known issue" for the Ubuntu folks where they include some GCC LTO flags by default, knowing that it will break a Clang build. So now Swift 5.8.1 is in Ubuntu Plucky and should be available to all Ubuntu derivatives.

Swift 5.9.2 is ready now and should be on its way into the repositories soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A feature request or implementation platform support
Projects
None yet
Development

No branches or pull requests