Add Android community CI #55
Conversation
|
@marcprux Email sent for additional information. Thanks |
|
The main issue with the current setup is that it was using the bootstrap build, so once that broke, it no longer built the compiler. If you simply use the same build preset but provide an OS environment with a prebuilt Swift toolchain and Android NDK 26d available, it should work. |
Yes, I expect some tweaks will be needed to get it going again. Once the builds start getting submitted to the machine I'll start filing PRs to get the preset working. I submitted the requested information a couple weeks ago, but haven't heard anything since. |
|
Btw, I updated trunk to work with NDK 27, so the latest NDK 27c should work now. |
|
The NDK 27c installation is at Tacking it onto the end of the command with Should I make a PR for build-presets.ini with this flag, or should be added by the Jenkins controller? |
|
Manually running the Android build with: eventually errors out with: This is the same error cited by swiftlang/swift#77795, so I'm wondering if swiftlang/swift#77815 will fix it. |
The latter, Swift build presets are customizable with such declared command-line flags, though I believe it ignores the
That shouldn't be necessary: if you simply install a prebuilt Swift toolchain for the host and add it to the There is no reason to bootstrap the trunk compiler on this CI when there are already prebuilt host toolchains available for linux. |
That seems to not be the case: |
|
You snipped out the relevant portion where the CMake config for the trunk Swift compiler is supposed to find it: That log snippet shows how the linux CI has CMake find the prebuilt Swift 5.8.1 toolchain used from its Is it possible your |
|
The Jenkins-submitted build is still failing immediately due to the missing ndk_path (https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-24.04-android-build/lastBuild/consoleFull). But when I manually add it to the PATH and run the build manually, it does seem to pick up the compiler: @shahmishal, can you configure Jenkins for the Android CI to:
|
@shahmishal, should I be requesting this through some other forum? |
|
@marcprux Sorry, I was out last week. Let me look into this later today. |
|
Both jobs are now updated: |
|
OK, it finds the prebuilt Swift compiler now, so you can start updating the presets. You'll want to start building the swift-syntax parser again, add @drodriguez, is it okay if we modify the README to remove the links to your old community CI, which it looks like stopped running a couple months ago? |
Yes, we should update the README. |
Oh wait, you are proposing CI for aarch64 and x86_64 instead. In that case, maybe just leave the base preset alone for that variable, so it defaults to aarch64, then add a new x86_64 preset and update this pull to explicitly list that new x86_64 preset instead. |
Even though the build is now finding the host toolchain: It is still ultimately failing with the same error: Should we be adding I almost wonder if we should add a new Android preset that exactly mirrors your working build script: |
That is most likely related to the now incorrect
The point of this community CI is to regularly build the trunk compiler and test it for Android, so that wouldn't make sense. I do think the official CI rebuilds way too much code, and told Mishal years ago that it should employ some build caching, so that a recently built compiler build directory for a certain compiler commit can be reused between, let's say, swift-foundation builds. Some of that has been done, eg SwiftPM "self-hosted" smoke tests now use the prebuilt nightly snapshot compiler instead of building the latest trunk compiler commit from source each time, but not nearly enough build caching on the CI has been added. The problem is that you'd have to clearly state the build dependency tree first to know what repo depends on what, eg the compiler build also builds libdispatch so that's a circular dependency of sorts, and that work likely hasn't been done. I hope Mishal gets some help with his official CI work- I thought I saw on Mastodon that Apple was hiring build engineers for Swift- and they can get that implemented some day.
That's the plan for the official CI at some point, which is moving up to the top of my task queue, though it would have to build the trunk compiler also, unlike my Android CI which just uses the prebuilt compiler snapshots. If you find setting this up too much of a hassle, we could just wait for that official Android CI instead. This preset will only really help keep the compiler validation suite up to date for Android, which has little to no effect on my Android CI and its SDK bundles. You mentioned some failed trunk runs on my Android CI in asking for this community CI, but that's mostly related to my upstreaming Android patches for other repos like SwiftPM, which aren't even built by this preset. And considering I expect to get the official Android CI submitted in the next couple months, this community CI would only be useful for those few prior months. Your call: there is some value in fixing the last tests in the compiler validation suite for Android, and getting this preset working isn't much effort. But this community CI will have basically no effect on my own Android CI and the trunk SDK bundles it generates. |
What form will this take? Will it be just another preset that is built using the same Jenkins process, or a whole separate CI mechanism? If the latter, how do you envision it integrating with PRs and the like? I certainly don't want to duplicate work. My goal was to first get the current preset working, and then start proposing additions to it like building SwiftPM, Foundation, and other essential packages that frequently need to be patched due to breakage. |
|
@marcprux the existing Android presets were the presets used by the community CI which was disconnected at the end of September. Feel free to repourpose those presets for the new community CI. No need to create new ones (well, if you want to modernize to skip armv7 and do something more meaningful that should be fine). If a Swift toolchain is present in the new CI machines, the changes from swiftlang/swift#68895 should not be necessary anymore. |
It would be part of the official CI, which are Docker images that run a build preset through Jenkins, similar to this community CI.
Initially, it would be another post-commit CI, like the couple dozen that run now. Maybe it would graduate to a pre-commit CI at some point, as Alex discussed on the forum earlier this year.
You wouldn't, I haven't done anything about the official Android CI yet. If you want to spearhead this, whether with the community or official CI, go for it, and I will assist when I can.
I agree about the corelibs, as I wrote up in my plan for the official CI when discussing it with Alex on the forum. As for building SwiftPM and other outside packages for Android, I don't know if the official CI does that type of extra validation, particularly for platforms for which it does not distribute an official native toolchain, only a cross-compilation SDK. Building the compiler and corelibs and running their tests is mostly what I see it doing for other platforms now. Also, in my experience, everything has been pretty stable and I see almost no breakage with most of the Swift repos. If you're referring to the sometimes failing builds on my Android CI, that's usually because I'm still upstreaming patches for the new Android overlay and Foundation rewrite from earlier this year and I don't know when those will be included in a new snapshot tag. So when one of my patches makes it into a snapshot tag, my CI will try to apply the same patch and fail. That's not a big deal and is quickly fixed, plus it's just a temporary situation after all the Android overlay/rewrite changes earlier this year. |
|
Alright, changing the base preset and adding one for x86_64 shows that this CI works, as it now builds the Swift compiler and fails when cross-compiling the stdlib for Android, because the passed-in path to the Android NDK on this CI doesn't have the NDK version. That's fine: I'll update the stdlib CMake config to check the NDK's So keep this NDK path the way it is, but update this pull to use the right aarch64 and new x86_64 presets now. @shahmishal, I don't know if you need to change any other internal config to use the new x86_64 preset also, but once those last changes are in, we can get this merged and Marc and I can then iterate on the presets and fixing tests for Android. |
|
If we need to change the preset, please update the |
I've updated the x86_64 preset to match the new entry in build-presets.ini. That should be all that is needed, yes? The aarch64 preset is unchanged. |
|
@shahmishal, we got the compiler validation suite running on this new CI finally, so once you adjust the CI jobs to these two presets, this is ready for merging. |
|
Oh, one discrepancy I noticed is that this pull says 22.04, but the CI jobs are named and seem to run 24.04. |
The previous Android CI has been offline for nearly a year. We'd like to volunteer to take on the community CI builds for Android.