rustc: Store metadata-in-rlibs in object files #84449

alexcrichton · 2021-04-22T20:24:27Z

This commit updates how rustc compiler metadata is stored in rlibs.
Previously metadata was stored as a raw file that has the same format as
--emit metadata. After this commit, however, the metadata is encoded
into a small object file which has one section which is the contents of
the metadata.

The motivation for this commit is to fix a common case where #83730
arises. The problem is that when rustc crates a dylib crate type it
needs to include entire rlib files into the dylib, so it passes
--whole-archive (or the equivalent) to the linker. The problem with
this, though, is that the linker will attempt to read all files in the
archive. If the metadata file were left as-is (today) then the linker
would generate an error saying it can't read the file. The previous
solution was to alter the rlib just before linking, creating a new
archive in a temporary directory which has the metadata file removed.

This problem from before this commit is now removed if the metadata file
is stored in an object file that the linker can read. The only caveat we
have to take care of is to ensure that the linker never actually
includes the contents of the object file into the final output. We apply
similar tricks as the .llvmbc bytecode sections to do this.

This involved changing the metadata loading code a bit, namely updating
some of the LLVM C APIs used to use non-deprecated ones and fiddling
with the lifetimes a bit to get everything to work out. Otherwise though
this isn't intended to be a functional change really, only that metadata
is stored differently in archives now.

This should end up fixing #83730 because by default dylibs will no
longer have their rlib dependencies "altered" meaning that
split-debuginfo will continue to have valid paths pointing at the
original rlibs. (note that we still "alter" rlibs if LTO is enabled to
remove Rust object files and we also "alter" for the #[link(cfg)]
feature, but that's rarely used).

Closes #83730

rust-highfive · 2021-04-22T20:24:30Z

r? @estebank

(rust-highfive has picked a reviewer for you, use r? to override)

compiler/rustc_codegen_llvm/src/metadata.rs

compiler/rustc_codegen_ssa/src/back/link.rs

bjorn3 · 2021-04-26T13:44:13Z

This will need rustc_codegen_cranelift to be updated to do the same thing. Otherwise it becomes impossible to link against a cg_llvm compiled standard library.

alexcrichton · 2021-05-04T23:30:28Z

@nagisa would you be up for reviewing this?

nagisa · 2021-05-04T23:53:48Z

r? @nagisa Will see about getting to this on Friday/weekend.

But so far a couple of remarks:

The problem with this, though, is that the linker will attempt to read all files in the archive. If the metadata file were left as-is (today) then the linker would generate an error saying it can't read the file.

Was this radared to apple at any point? Its not unreasonable to have non-object code in archives that are used for linking. Not that it needs to block this PR of course.

It was convenient to some extent that rmeta stored within the rlib archive and rmeta output by --emit=metadata used a similar format. You could use the same tooling to inspect either without further effort. If we're doing this maybe it makes sense to adjust the rmeta file format to be a metadata-in-object too? If it hasn't been already, that is.

@bors try @rust-timer queue

rust-timer · 2021-05-04T23:53:50Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2021-05-04T23:53:58Z

⌛ Trying commit 5cb3a4815eec44805bb2f7e089285f6172d86517 with merge 7e38754f1942de640d8f2e8b14b892b617252f70...

alexcrichton · 2021-05-05T00:39:34Z

No rush of course!

I haven't reported any issues to other linkers, but AFAIK the behavior of "fail the link if anything doesn't look like an object file" is present in all linkers (link.exe, gnu ld, lld, macOS ld, etc)

As for --emit metadata I'm happy to implement the change for that as well, it should be easy enough. Alternatively I can document some commands to extract the metadata from the object file if you think that would be useful.

bors · 2021-05-05T00:54:21Z

☀️ Try build successful - checks-actions
Build commit: 7e38754f1942de640d8f2e8b14b892b617252f70 (7e38754f1942de640d8f2e8b14b892b617252f70)

rust-timer · 2021-05-05T00:54:22Z

Queued 7e38754f1942de640d8f2e8b14b892b617252f70 with parent ae8b84b, future comparison URL.

rust-timer · 2021-05-05T03:14:33Z

Finished benchmarking try commit (7e38754f1942de640d8f2e8b14b892b617252f70): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

Cargo.lock

compiler/rustc_codegen_ssa/src/back/link.rs

compiler/rustc_codegen_llvm/src/metadata.rs

compiler/rustc_codegen_ssa/src/back/link.rs

alexcrichton · 2021-05-11T14:50:15Z

I'm gonna hold off on trying to fix the CI errors until #83640 is merged.

alexcrichton · 2021-05-13T19:16:11Z

I went ahead and rebased on #83640 as it fixed the errors locally. I also deleted a bunch of LLVM bindings bits which we no longer use as a result (e.g. parsing object files and dealing with memory buffers)

bors · 2021-06-04T19:29:55Z

⌛ Testing commit 0e03387 with merge c79419a...

bors · 2021-06-04T22:18:11Z

☀️ Test successful - checks-actions
Approved by: nagisa
Pushing c79419a to master...

alexcrichton · 2021-06-04T22:21:17Z

Thanks @philipc for your help in the object crate to get this landed!

The code intended to set the IMAGE_SCN_LNK_REMOVE flag for the .rmeta section, however the value of this flag was set to zero. Instead use the actual value provided by the object crate. This dates back to the original introduction of this code in PR rust-lang#84449, so we were never setting this flag. As I'm not on Windows, I'm not sure whether that means we were embedding .rmeta into executables, or whether the section ended up getting stripped for some other reason.

Actually set IMAGE_SCN_LNK_REMOVE for .rmeta The code intended to set the IMAGE_SCN_LNK_REMOVE flag for the .rmeta section, however the value of this flag was set to zero. Instead use the actual value provided by the object crate. This dates back to the original introduction of this code in PR rust-lang#84449, so we were never setting this flag. As I'm not on Windows, I'm not sure whether that means we were embedding .rmeta into executables, or whether the section ended up getting stripped for some other reason.

The goal of this commit is to remove warnings using LLVM tip-of-tree `wasm-ld`. In llvm/llvm-project#78658 the `wasm-ld` LLD driver no longer looks at archive indices and instead looks at all the objects in archives. Previously `lib.rmeta` files were simply raw rustc metadata bytes, not wasm objects, meaning that `wasm-ld` would emit a warning indicating so. WebAssembly targets previously passed `--fatal-warnings` to `wasm-ld` by default which meant that if Rust were to update to LLVM 18 then all wasm targets would not work. This immediate blocker was resolved in rust-lang#120278 which removed `--fatal-warnings` which enabled a theoretical update to LLVM 18 for wasm targets. This current state is ok-enough for now because rustc squashes all linker output by default if it doesn't fail. This means, for example, that rustc squashes all the linker warnings coming out of `wasm-ld` about `lib.rmeta` files with LLVM 18. This again isn't a pressing issue because the information is all hidden, but it runs the risk of being annoying if another linker error were to happen and then the output would have all these unrelated warnings that couldn't be fixed. Thus, this PR comes into the picture. The goal of this PR is to resolve these warnings by using the WebAssembly object file format on wasm targets instead of using raw rustc metadata. When I first implemented the rlib-in-objects scheme in rust-lang#84449 I remember either concluding that `wasm-ld` would either include the metadata in the output or I thought we didn't have to do anything there at all. I think I was wrong on both counts as `wasm-ld` does not include the metadata in the final output unless the object is referenced and we do actually need to do something to resolve these warnings. This PR updates the object file format containing rustc metadata on WebAssembly targets to be an actual WebAssembly file. To avoid bringing in any new dependencies I've opted to hand-code this encoding at this time. If the object gets more complicated though it'd probably be best to pull in `wasmparser` and `wasm-encoder`. For now though there's two adjacent functions reading/writing wasm. The only caveat I know of with this is that if `wasm-ld` does indeed look at the object file then the metadata will be included in the final output. I believe the only thing that could cause that at this time is `--whole-archive` which I don't think is passed for rlibs. I would clarify that I'm not 100% certain about this, however.

The goal of this commit is to remove warnings using LLVM tip-of-tree `wasm-ld`. In llvm/llvm-project#78658 the `wasm-ld` LLD driver no longer looks at archive indices and instead looks at all the objects in archives. Previously `lib.rmeta` files were simply raw rustc metadata bytes, not wasm objects, meaning that `wasm-ld` would emit a warning indicating so. WebAssembly targets previously passed `--fatal-warnings` to `wasm-ld` by default which meant that if Rust were to update to LLVM 18 then all wasm targets would not work. This immediate blocker was resolved in rust-lang#120278 which removed `--fatal-warnings` which enabled a theoretical update to LLVM 18 for wasm targets. This current state is ok-enough for now because rustc squashes all linker output by default if it doesn't fail. This means, for example, that rustc squashes all the linker warnings coming out of `wasm-ld` about `lib.rmeta` files with LLVM 18. This again isn't a pressing issue because the information is all hidden, but it runs the risk of being annoying if another linker error were to happen and then the output would have all these unrelated warnings that couldn't be fixed. Thus, this PR comes into the picture. The goal of this PR is to resolve these warnings by using the WebAssembly object file format on wasm targets instead of using raw rustc metadata. When I first implemented the rlib-in-objects scheme in rust-lang#84449 I remember either concluding that `wasm-ld` would either include the metadata in the output or I thought we didn't have to do anything there at all. I think I was wrong on both counts as `wasm-ld` does not include the metadata in the final output unless the object is referenced and we do actually need to do something to resolve these warnings. This PR updates the object file format containing rustc metadata on WebAssembly targets to be an actual WebAssembly file. This enables the `wasm` feature of the `object` crate to be able to read the custom section in the same manner as other platforms, but currently `object` doesn't support writing wasm object files so a handwritten encoder is used instead. The only caveat I know of with this is that if `wasm-ld` does indeed look at the object file then the metadata will be included in the final output. I believe the only thing that could cause that at this time is `--whole-archive` which I don't think is passed for rlibs. I would clarify that I'm not 100% certain about this, however.

…wesleywiser wasm: Store rlib metadata in wasm object files The goal of this commit is to remove warnings using LLVM tip-of-tree `wasm-ld`. In llvm/llvm-project#78658 the `wasm-ld` LLD driver no longer looks at archive indices and instead looks at all the objects in archives. Previously `lib.rmeta` files were simply raw rustc metadata bytes, not wasm objects, meaning that `wasm-ld` would emit a warning indicating so. WebAssembly targets previously passed `--fatal-warnings` to `wasm-ld` by default which meant that if Rust were to update to LLVM 18 then all wasm targets would not work. This immediate blocker was resolved in rust-lang#120278 which removed `--fatal-warnings` which enabled a theoretical update to LLVM 18 for wasm targets. This current state is ok-enough for now because rustc squashes all linker output by default if it doesn't fail. This means, for example, that rustc squashes all the linker warnings coming out of `wasm-ld` about `lib.rmeta` files with LLVM 18. This again isn't a pressing issue because the information is all hidden, but it runs the risk of being annoying if another linker error were to happen and then the output would have all these unrelated warnings that couldn't be fixed. Thus, this PR comes into the picture. The goal of this PR is to resolve these warnings by using the WebAssembly object file format on wasm targets instead of using raw rustc metadata. When I first implemented the rlib-in-objects scheme in rust-lang#84449 I remember either concluding that `wasm-ld` would either include the metadata in the output or I thought we didn't have to do anything there at all. I think I was wrong on both counts as `wasm-ld` does not include the metadata in the final output unless the object is referenced and we do actually need to do something to resolve these warnings. This PR updates the object file format containing rustc metadata on WebAssembly targets to be an actual WebAssembly file. To avoid bringing in any new dependencies I've opted to hand-code this encoding at this time. If the object gets more complicated though it'd probably be best to pull in `wasmparser` and `wasm-encoder`. For now though there's two adjacent functions reading/writing wasm. The only caveat I know of with this is that if `wasm-ld` does indeed look at the object file then the metadata will be included in the final output. I believe the only thing that could cause that at this time is `--whole-archive` which I don't think is passed for rlibs. I would clarify that I'm not 100% certain about this, however.

The goal of this commit is to remove warnings using LLVM tip-of-tree `wasm-ld`. In llvm/llvm-project#78658 the `wasm-ld` LLD driver no longer looks at archive indices and instead looks at all the objects in archives. Previously `lib.rmeta` files were simply raw rustc metadata bytes, not wasm objects, meaning that `wasm-ld` would emit a warning indicating so. WebAssembly targets previously passed `--fatal-warnings` to `wasm-ld` by default which meant that if Rust were to update to LLVM 18 then all wasm targets would not work. This immediate blocker was resolved in rust-lang#120278 which removed `--fatal-warnings` which enabled a theoretical update to LLVM 18 for wasm targets. This current state is ok-enough for now because rustc squashes all linker output by default if it doesn't fail. This means, for example, that rustc squashes all the linker warnings coming out of `wasm-ld` about `lib.rmeta` files with LLVM 18. This again isn't a pressing issue because the information is all hidden, but it runs the risk of being annoying if another linker error were to happen and then the output would have all these unrelated warnings that couldn't be fixed. Thus, this PR comes into the picture. The goal of this PR is to resolve these warnings by using the WebAssembly object file format on wasm targets instead of using raw rustc metadata. When I first implemented the rlib-in-objects scheme in rust-lang#84449 I remember either concluding that `wasm-ld` would either include the metadata in the output or I thought we didn't have to do anything there at all. I think I was wrong on both counts as `wasm-ld` does not include the metadata in the final output unless the object is referenced and we do actually need to do something to resolve these warnings. This PR updates the object file format containing rustc metadata on WebAssembly targets to be an actual WebAssembly file. This enables the `wasm` feature of the `object` crate to be able to read the custom section in the same manner as other platforms, but currently `object` doesn't support writing wasm object files so a handwritten encoder is used instead. The only caveat I know of with this is that if `wasm-ld` does indeed look at the object file then the metadata will be included in the final output. I believe the only thing that could cause that at this time is `--whole-archive` which I don't think is passed for rlibs. I would clarify that I'm not 100% certain about this, however.

…sleywiser,bjorn3 wasm: Store rlib metadata in wasm object files The goal of this commit is to remove warnings using LLVM tip-of-tree `wasm-ld`. In llvm/llvm-project#78658 the `wasm-ld` LLD driver no longer looks at archive indices and instead looks at all the objects in archives. Previously `lib.rmeta` files were simply raw rustc metadata bytes, not wasm objects, meaning that `wasm-ld` would emit a warning indicating so. WebAssembly targets previously passed `--fatal-warnings` to `wasm-ld` by default which meant that if Rust were to update to LLVM 18 then all wasm targets would not work. This immediate blocker was resolved in rust-lang#120278 which removed `--fatal-warnings` which enabled a theoretical update to LLVM 18 for wasm targets. This current state is ok-enough for now because rustc squashes all linker output by default if it doesn't fail. This means, for example, that rustc squashes all the linker warnings coming out of `wasm-ld` about `lib.rmeta` files with LLVM 18. This again isn't a pressing issue because the information is all hidden, but it runs the risk of being annoying if another linker error were to happen and then the output would have all these unrelated warnings that couldn't be fixed. Thus, this PR comes into the picture. The goal of this PR is to resolve these warnings by using the WebAssembly object file format on wasm targets instead of using raw rustc metadata. When I first implemented the rlib-in-objects scheme in rust-lang#84449 I remember either concluding that `wasm-ld` would either include the metadata in the output or I thought we didn't have to do anything there at all. I think I was wrong on both counts as `wasm-ld` does not include the metadata in the final output unless the object is referenced and we do actually need to do something to resolve these warnings. This PR updates the object file format containing rustc metadata on WebAssembly targets to be an actual WebAssembly file. To avoid bringing in any new dependencies I've opted to hand-code this encoding at this time. If the object gets more complicated though it'd probably be best to pull in `wasmparser` and `wasm-encoder`. For now though there's two adjacent functions reading/writing wasm. The only caveat I know of with this is that if `wasm-ld` does indeed look at the object file then the metadata will be included in the final output. I believe the only thing that could cause that at this time is `--whole-archive` which I don't think is passed for rlibs. I would clarify that I'm not 100% certain about this, however.

Added for rust-lang/rust#84449

rust-highfive assigned estebank Apr 22, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 22, 2021

alexcrichton commented Apr 22, 2021

View reviewed changes

compiler/rustc_codegen_llvm/src/metadata.rs Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

alexcrichton force-pushed the metadata-in-object branch from 5dfa196 to 6116148 Compare April 22, 2021 21:37

This comment has been minimized.

Sign in to view

bjorn3 reviewed Apr 26, 2021

View reviewed changes

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

rust-highfive assigned nagisa and unassigned estebank May 4, 2021

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 4, 2021

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 5, 2021

nagisa reviewed May 7, 2021

View reviewed changes

Cargo.lock Show resolved Hide resolved

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

compiler/rustc_codegen_llvm/src/metadata.rs Outdated Show resolved Hide resolved

bjorn3 mentioned this pull request May 7, 2021

Use the object crate for metadata reading #83640

Merged

nagisa reviewed May 9, 2021

View reviewed changes

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

alexcrichton force-pushed the metadata-in-object branch from 5cb3a48 to 85747a7 Compare May 10, 2021 14:06

This comment has been minimized.

Sign in to view

alexcrichton force-pushed the metadata-in-object branch from 85747a7 to 6378706 Compare May 13, 2021 19:14

bors added the merged-by-bors This PR was explicitly merged by bors. label Jun 4, 2021

bors merged commit c79419a into rust-lang:master Jun 4, 2021

rustbot added this to the 1.54.0 milestone Jun 4, 2021

alexcrichton deleted the metadata-in-object branch June 4, 2021 22:18

This was referenced Jun 4, 2021

Partial support for raw-dylib linkage #84171

Merged

linker: Never use whole-archive linking unless explicitly requested #85144

Closed

nikic mentioned this pull request Dec 19, 2021

Actually set IMAGE_SCN_LNK_REMOVE for .rmeta #92107

Merged

davidhewitt mentioned this pull request Jan 3, 2022

rustc_driver.dll size on x86_64-pc-windows-gnu toolchain #92539

Closed

ehuss mentioned this pull request Oct 14, 2022

rlib-format-packed-bundled-libs test failures on Xcode 14 #103044

Closed

bjorn3 mentioned this pull request Nov 23, 2022

Update use of libc::timespec to prepare for future libc version Amanieu/parking_lot#363

Merged

ehuss mentioned this pull request Dec 2, 2022

XCode 14 and older rustup installed rustc versions are incompatible #105167

Closed

Cldfire mentioned this pull request Jan 21, 2023

Build in CI on macOS too Cldfire/nvml-wrapper#42

Merged

alexcrichton mentioned this pull request Feb 2, 2024

wasm: Store rlib metadata in wasm object files #120588

Merged

mcbegamerxx954 pushed a commit to mcbegamerxx954/object that referenced this pull request Jun 15, 2024

Add basic support for mips64/powerpc/powerpc64/riscv64

a3761e5

Added for rust-lang/rust#84449

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Store metadata-in-rlibs in object files #84449

rustc: Store metadata-in-rlibs in object files #84449

alexcrichton commented Apr 22, 2021

rust-highfive commented Apr 22, 2021

This comment has been minimized.

This comment has been minimized.

bjorn3 commented Apr 26, 2021

alexcrichton commented May 4, 2021

nagisa commented May 4, 2021

rust-timer commented May 4, 2021

bors commented May 4, 2021

alexcrichton commented May 5, 2021

bors commented May 5, 2021

rust-timer commented May 5, 2021

rust-timer commented May 5, 2021

This comment has been minimized.

alexcrichton commented May 11, 2021

alexcrichton commented May 13, 2021

bors commented Jun 4, 2021

bors commented Jun 4, 2021

alexcrichton commented Jun 4, 2021

rustc: Store metadata-in-rlibs in object files #84449

rustc: Store metadata-in-rlibs in object files #84449

Conversation

alexcrichton commented Apr 22, 2021

rust-highfive commented Apr 22, 2021

This comment has been minimized.

This comment has been minimized.

bjorn3 commented Apr 26, 2021

alexcrichton commented May 4, 2021

nagisa commented May 4, 2021

rust-timer commented May 4, 2021

bors commented May 4, 2021

alexcrichton commented May 5, 2021

bors commented May 5, 2021

rust-timer commented May 5, 2021

rust-timer commented May 5, 2021

This comment has been minimized.

alexcrichton commented May 11, 2021

alexcrichton commented May 13, 2021

bors commented Jun 4, 2021

bors commented Jun 4, 2021

alexcrichton commented Jun 4, 2021