Bootstrap fails due to system LLVM libffi link error

[MagaTailor]

Trying to bootstrap rust on aarch64 Linux using --llvm-root fails:

error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc.0.o" "-o" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/usr/lib/llvm-3.8/lib" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_driver-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_passes-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_borrowck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_lint-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_resolve-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_typeck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_plugin-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_privacy-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_trans-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_platform_intrinsics-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_incremental-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax_ext-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_mir-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_eval-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_metadata-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_save_analysis-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_math-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "getopts-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "flate-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_back-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rbml-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_data_structures-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "serialize-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "log-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "term-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "fmt_macros-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "arena-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_llvm-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "graphviz-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "std-fe3cdf61" "-l" "rt" "-l" "dl" "-l" "tinfo" "-l" "pthread" "-l" "z" "-l" "m" "-l" "stdc++" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-Wl,-rpath,$ORIGIN/../lib" "-Wl,-rpath,/usr/local/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,--enable-new-dtags" "-s" "-l" "compiler-rt"

note: ../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_float'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint64'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_pointer'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint32'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_void'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint8'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_call'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint16'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_double'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_prep_cif'
collect2: error: ld returned 1 exit status

error: aborting due to previous error
../mk/target.mk:220: recipe for target 'aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc' failed
make: *** [aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc] Error 101

For some reason the correct llvm configuration is not being detected and -lffi has to be passed manually for the bootstrap to succeed.

[brson]

In-tree, we configure LLVM without -lffi so maybe there's a discrepancy when we encounter an LLVM with it compiled in.

[MagaTailor]

Yup, the LLVM I was bootsrapping against was clearly built with LLVM_ENABLE_FFI.

[jethrogb]

Running into the same isse. Shouldn't llvm-config output this requirement?

Btw for anyone reading this: a quick fix is to add:

#[link(name = "ffi")] extern {}

to src/librustc_llvm/lib.rs.

[japaric]

I'm also experiencing this on x86_64 Ubuntu 16.04 using the llvm-dev package and --llvm-root.

[MagaTailor]

The title should be edited then, and btw, the quickest non-permanent solution looks like this:
RUSTFLAGS='-C link-args=-lffi'

[japaric]

It seems this is a duplicate of #29694

[japaric]

Recently, I have been annoyed by this a lot. I've been boostraping rustc against system llvm a lot in different Docker containers and I have to remember to apply @jethrogb patch/fix to every rust branch/worktree I work on and then I have to be careful to not commit this fix to the branch I'm working on. 😞

Could we simply commit @jethrogb's fix into master? I tested bootstrapping rust against our llvm fork with that fix merged into master (5a5736d) and at least, on Linux, it bootstraped without problems.

P.S. @petevine's RUSTFLAGS solution didn't work for me.

[MagaTailor]

@japaric That's all there's to it, and it definitely works for the Makefiles bootstrap.

[alexcrichton]

This is of course not intentional, and fixes are always welcome! The trick here is detecting whether we're building against an LLVM that needs libffi. For example the in-tree LLVM doesn't and we shouldn't link to it as a result.

[cuviper]

FWIW, Fedora's LLVM is built with FFI, but this is usually fine because libLLVM.so links to libffi.so itself. However, with llvm-static.rpm installed, llvm-config starts giving static link options (but not -lffi), and ldd -r librustc_llvm-*.so indeed shows a few undefined symbol: ffi_....

[cuviper]

IOW I blame llvm-config, not that blame solves anything.
(I also don't like that it doesn't let me choose shared linkage even when static libraries are present...)

[userwithuid]

@cuviper Regarding 'choose shared over static llvm':
If I understand correctly, since 3.9 llvm-config will prefer the combo-dynlib by default over any static libs present. And even if you don't have the combo, but say shared and static component libs installed, there are now options to specifically choose --link-static or --link-shared.

Relevant llvm commit explains it: llvm-mirror/llvm@93083d4

Doesn't fix this bug here, external llvm + ffi enabled + intentionally linking llvm static libs still triggers it, just thought it would be nice to know.

(BTW, the ffi-issue seems to be sort of a known shortcoming of llvm-config --system-libs. It only statically lists the LLVMSupport sysdeps, but ffi is the (currently only?) sysdep from another component - LLVMExecutionEngine)

[cuviper]

@userwithuid Thanks, I wasn't aware of that! I'll try that out, and maybe we should expose those explicit link options somehow too.

[ishitatsuyuki]

Please tell me the current situation; rustbuild is enabled by default, the old workarounds may not work.

[MagaTailor]

Yeah, last time I tried it didn't even build due to some completely unfamiliar link errors. (edit: I meant exactly this)

If it's still a problem in the new build system, please open a more specific issue.

[ishitatsuyuki]

I'm quite sure that this is still a problem; please don't blindly close it.

[jethrogb]

Yes this should remain open. @MagaTailor I don't understand what you mean, this issue is quite specific.

[ht990332]

I went ahead and filed a new bug 39880.

[ht990332]

@csmoe Is this an issue again in post 1.31.1 releases? I have version 1.31.1 installed and did not see this issue when compiling rust against system llvm 7.0.1.

[cuviper]

@Hussamt IIRC it was only an issue with linking external LLVM statically, and even then only if that build is configured with -DLLVM_ENABLE_FFI:BOOL=ON. At least on Fedora rawhide, I still get:

$ rpm -q llvm
llvm-7.0.1-1.fc30.x86_64
$ llvm-config --link-static --system-libs
-lz -lrt -ldl -ltinfo -lpthread -lm

... missing -lffi, even though some of the static libs do use it. But this is really an LLVM problem, as noted in #34486 (comment), so I'm not sure we need this Rust issue open except as a "known issue, see elsewhere" tracker.

Our official Fedora rust builds are linked dynamically (--enable-llvm-link-shared), and this is fine.

[jyn514]

Triage: this seems to still be an issue today: https://github.com/llvm-mirror/llvm/blob/2c4ca6832fa6b306ee6a7010bfb80a3f2596f824/tools/llvm-config/llvm-config.cpp#L705

[mati865]

@jyn514 FYI that linked comment is not the problem.
The problem is LLVM doesn't add libffi to LLVM_SYSTEM_LIBS.
Some time ago I have fixed libz3 which had the same issue: https://reviews.llvm.org/D85195 but libffi is not that easy to fix. Somebody could use it as an guidance on fixing libffi though.