Error cargo building std as a dylib

[japaric]

Or should cargo building std only produce a .rlib?

Now that #35021 landed and compiler-rt intrinsics are built as part of the std crate dependencies,
one can (cross) compile binaries if the alloc_system (or alloc_jemalloc), panic_unwind and
std crates are listed under the crate Cargo.toml:

$ cargo new --bin hello

$ cd hello

$ edit src/main.rs && cat src/main.rs
#![feature(alloc_system)]

extern crate alloc_system;

fn main() {
    println!("Hello, world!");
}

$ edit Cargo.toml && cat Cargo.toml
[package]
name = "hello"

[dependencies]
alloc_syste = { path = "$(rustc --print sysroot)/lib/rustlib/src/rust/src/liballoc_system" }
panic_unwind = { path = "$(rustc --print sysroot)/lib/rustlib/src/rust/src/libpanic_unwind" }
std = { path = "$(rustc --print sysroot)/lib/rustlib/src/rust/src/libstd" }

# if alloc_jemalloc is used
# [features]
# default = ["std/jemalloc"]

$ cargo build --target $CROSS

if and only if the Cargo.toml of the std crate is modified as follows

 [lib]
 name = "std"
 path = "lib.rs"
-crate-type = ["dylib", "rlib"]

 [dependencies]
 alloc = { path = "../liballoc" }

to not build a dylib. Without this change cargo building hello fails with:

$ cargo build --target $CROSS
     Running `rustc $sysroot/lib/rustlib/src/rust/src/libstd/lib.rs --crate-name std --crate-type dylib --crate-type rlib -C prefer-dynamic -g -C metadata=7a8ef155232f363e --out-dir $PWD/target/$CROSS/debug/deps --emit=dep-info,link --target $CROSS -L dependency=$PWD/target/$CROSS/debug/deps --extern core=$PWD/target/$CROSS/debug/deps/libcore.rlib --extern compiler_builtins=$PWD/target/$CROSS/debug/deps/libcompiler_builtins.rlib --extern rand=$PWD/target/$CROSS/debug/deps/librand.rlib --extern panic_abort=$PWD/target/$CROSS/debug/deps/libpanic_abort.rlib --extern libc=$PWD/target/$CROSS/debug/deps/liblibc.rlib --extern rustc_unicode=$PWD/target/$CROSS/debug/deps/librustc_unicode.rlib --extern alloc=$PWD/target/$CROSS/debug/deps/liballoc.rlib --extern alloc_system=$PWD/target/$CROSS/debug/deps/liballoc_system.rlib --extern collections=$PWD/target/$CROSS/debug/deps/libcollections.rlib --extern panic_unwind=$PWD/target/$CROSS/debug/deps/libpanic_unwind.rlib --extern unwind=$PWD/target/$CROSS/debug/deps/libunwind.rlib --cfg cargobuild -l dl -l rt -l pthread -L native=$PWD/target/$CROSS/debug/build/compiler_builtins-31a1189c65ac3d55/out`
warning: ar: `u' modifier ignored since `D' is the default (see `U')
   Compiling std v0.0.0 (file://$sysroot/lib/rustlib/src/rust/src/libstd)
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

error: Could not compile `std`.

during the compilation of the std crate. (On that note, I'm sure why building std as a dylib
fails? Is it because rustc injects a extern crate alloc_jemalloc into std source when compiling
a dylib but alloc_jemalloc is an optional dependency of std that's not build (the feature is
disabled) in this example?)

I discussed this with @alexcrichton and we were wondering if we should modify the std crate to only
compile the rlib to make this case work out of the box. And then hack bootstrap to force it to build
std as dylib because it is needed to link rustc.

cc @Ericson2314 I don't how/if this change would affect your Cargo+std RFC. (Probably not?)

[Ericson2314]

Looks good to me! (I hope once make is gone we can put things in their normal locations and the entire [lib] section can be elided.)

IIUC, we eventually want cargo to be smart enough to build deps (and later still, only public deps) as dylibs. This would be the decision of the bottom crate / builder not any library, so hacking rustbuild for now is a step in the right direction.

All these changes anticipate what I want to do once my RFC is merged, so keep 'em coming!

[alexcrichton]

For this specific error, I wonder if perhaps a more targeted fix would suffice? It looks like it's because jemalloc isn't compiled which the the target the standard library is being compiled for assumes will exist.

Basically, on this line, we'd link alloc_system for both stage0 and cfg(not(feature = "jemalloc")).

This does have interesting implications, though, for jemalloc. Does mean that building std in a custom fashion is a bit harder.

[Ericson2314]

mmm, how's that better? Custom stds may indeed want jemalloc, and I don't see what crate-type gets us long term.

[alexcrichton]

The problem is that when the compiler builds a dylib it attempts to link the default dylib allocator. For many targets that's jemalloc, but jemalloc isn't compiled unless you enable a feature.

[japaric]

The problem is that when the compiler builds a dylib it attempts to link the default dylib allocator. For many targets that's jemalloc, but jemalloc isn't compiled unless you enable a feature.

(Alternatively, we could default to system alloc everywhere. Less C is always good in my book)

[Ericson2314]

The "default dylib allocator" stuff is just a stopgap until we get general needs-provides, so I wouldn't sweat it.

[alexcrichton]

@japaric eventually hopefully!

[aidanhs]

The problem is that when the compiler builds a dylib it attempts to link the default dylib allocator. For many targets that's jemalloc, but jemalloc isn't compiled unless you enable a feature.

I read this and was confused, so to elaborate for other readers - dylibs are partitioned into 'dylibs used as rust dependencies' and 'other dylibs'. The former (the definition being encountered here) use the same allocator as the current rust compiler will give to executables, typically jemalloc. The latter is basically always the system allocator. See

rust/src/librustc_metadata/creader.rs

Line 822 in ff591b6

// * Binaries use jemalloc

-C prefer-dynamic is the way the former is indicated to rustc, and is (I think, more hazy here) detected in cargo by checking whether the crate is both a dylib and is not a member of the current workspace (i.e. is a dep somewhere).

[aidanhs]

Oh, and (again, just for the record) I think the issue here is relatively easily solved if your --target is a .json - I believe you just need to set "exe_allocation_crate" as "alloc_system". Unfortunately I'm not aware of a way to override parts of targets on the fly (I'd be delighted if I'm wrong!), so it's harder otherwise.

[alexcrichton]

I think this is likely fixed with jemalloc now removed in #55238 so closing