Add support for full RELRO #43170

kyrias · 2017-07-11T20:04:25Z

This commit adds support for full RELRO, and enables it for the
platforms I know have support for it.

Full RELRO makes the PLT+GOT data read-only on startup, preventing it
from being overwritten.

http://tk-blog.blogspot.com/2009/02/relro-not-so-well-known-memory.html

I'm not entirely certain if this is the best way to do it, but I figured mimicking the way it's done for PIE seemed like a good start at least. I'm not sure whether we want to have it enabled by default globally and then disabling it explicitly for targets that don't support it though. I'm also not sure whether the full_relro function should call bug!() or something like it for linkers that don't support it rather than no-opping.

This commit adds support for full RELRO, and enables it for the platforms I know have support for it. Full RELRO makes the PLT+GOT data read-only on startup, preventing it from being overwritten. http://tk-blog.blogspot.com/2009/02/relro-not-so-well-known-memory.html Fixes rust-lang#29877. Signed-off-by: Johannes Löthberg <[email protected]>

rust-highfive · 2017-07-11T20:04:31Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @eddyb (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

eddyb · 2017-07-11T20:08:01Z

r? @alexcrichton

cuviper · 2017-07-11T20:20:46Z

As I noted in #29877 (comment), I do add these flags on Fedora and EPEL7 (for RHEL7). However, I encountered bizarre linking bugs with this on RHEL6 for a few architectures -- I think that was on powerpc64 specifically. So that makes me really hesitate whether this should be the default behavior of rustc.

alexcrichton · 2017-07-11T21:13:52Z

Thanks for the PR @kyrias! This seems reasonable to me to try to set these by default, but @cuviper's concerns worry me! @cuviper you wouldn't happen still have any of those link error logs on hand, would you?

@kyrias do you know when these linker flags were introduced? One concern on the related issue is that not all linkers may support this, but my guess is that all recent-ish linkers should probably support it, just curious how far back it goes!

cuviper · 2017-07-11T21:43:49Z

I don't have those old logs anymore. I'm trying to reproduce it anew...

kyrias · 2017-07-12T12:54:24Z

GNU binutils seems to have added them in 2.16 which was released in 2005, and I'm not sure if it'd be worth it to support 12+ year old linkers. All Linux and BSD targets we want to support should be fine with this as it is. It's not enabled on e.g. Solaris though, since they use their own linker which doesn't seem to have support for RELRO.

alexcrichton · 2017-07-12T13:57:17Z

For comparison glibc 2.5 (the oldest one we support) was released on 2006-09-29, so I think we're probably covered here! Let's just wait and see if @cuviper's reproduction brings up something new, otherwise this seems fine to merge to me.

parched · 2017-07-12T17:43:33Z

Is there any worry about increased program load times with this?

kyrias · 2017-07-12T18:32:27Z

So, with full RELRO there is a slight performance impact when it resolves all the symbols, but in practice it's pretty much not noticeable unless the program has a very large number of symbols.

kyrias · 2017-07-12T18:36:14Z

It would possibly make sense to add a compiler flag to disable full BIND_NOW for specific packages though, so you'd end up with partial RELRO without the performance impact of BIND_NOW.

cuviper · 2017-07-12T19:46:32Z

Since Rust's kernel/glibc baseline is basically RHEL5, we can note that has binutils-2.17 too.

Also notable, since 2.27 ld defaults to enabling RELRO "for all Linux targets except FRV, HPPA, IA64 and MIPS." I don't think it defaults BIND_NOW though.

I'm still working on the reproducing my earlier issues. I couldn't get it manually, but I just found the git history of my package scripts where I turned it off, so I'm trying to build right before that again now.

cuviper · 2017-07-13T00:54:58Z

OK, not a link error, "just" a segfault:

running: "/builddir/build/BUILD/rust-1.17.0-powerpc64-unknown-linux-gnu/usr/bin/cargo" "build" "-j" "32" "--target" "powerpc64-unknown-linux-gnu" "-v" "--release" "--frozen" "--features" "panic-unwind backtrace" "--manifest-path" "/builddir/build/BUILD/rustc-1.17.0-src/src/libstd/Cargo.toml"
error: process didn't exit successfully: `/builddir/build/BUILD/rustc-1.17.0-src/build/bootstrap/debug/rustc -vV` (signal: 11, SIGSEGV: invalid memory reference)

I don't remember how I connected that to RELRO/BIND_NOW, and this attempt now was in the build farm so I don't actually have access to those failed artifacts. I'll start another build where I actually have a shell, using current master.

alexcrichton · 2017-07-13T14:28:20Z

@cuviper oh thanks for that! I wonder, does that mean that we should perhaps pass -z,relro but not -z,now? And maybe we shouldn't pass that on mips?

cuviper · 2017-07-14T00:21:12Z

I finally found my notes from before! We determined this to be a bug in the dynamic loader (ld.so), but sadly that bugzilla got marked private. I'll see if we can get a public bug opened...

I can still reproduce the issue while building 1.18, but 1.19-beta and later seem to run OK. I'm not especially confident in that though, given that the bug wasn't really rust's fault, so that "fix" may be a fluke.

kyrias · 2017-07-14T11:16:23Z

I think it would make sense to enable both for everything other than the ones binutils don't have it enabled for, and ppc64.

@cuviper Does it reproduce with partial RELRO as well, or just full RELRO?

cuviper · 2017-07-14T17:00:06Z

My bug just got closed as a duplicate of this one which is public:
https://bugzilla.redhat.com/show_bug.cgi?id=1398716

And indeed in rustc's case, it's a similar resolve_ifunc crash hitting gettimeofday before libc.so is relocated. I've asked for clarification about affected arches, as we may just be getting lucky that others are working.

@cuviper Does it reproduce with partial RELRO as well, or just full RELRO?

Using -z,relro alone works for me. It appears to be a problem only with -z,now / BIND_NOW. You can even induce it on existing rustc binaries by setting the environment LD_BIND_NOW=1.

kyrias · 2017-07-14T17:23:12Z

Any chance you could backport that commit to the glibc version you're using and seeing if that actually fixes it or not? If it does in fact fix it, I'm not sure what the best way forward would be. It would sort of suck to not be able to enable it due to a bug that got fixed 6 years ago, but if rustc wants to support RHEL5 as a baseline, which is 10 years old by now, I'm sort of uncertain.

I think it might be a good idea to have it on by default, and then have a built-time option for very out-of-date operating systems to disable it in that case.

Anyway, I could probably start by reworking this PR so that it treats partial and full RELRO separately, so that it would be easy to selectively disable full RELRO.

cuviper · 2017-07-14T18:01:35Z

Any chance you could backport that commit to the glibc version you're using and seeing if that actually fixes it or not? If it does in fact fix it, I'm not sure what the best way forward would be.

I have tested it, but I can't predict how soon an update may be released. Of course we have to be careful that it doesn't cause regressions elsewhere...

It would sort of suck to not be able to enable it due to a bug that got fixed 6 years ago, but if rustc wants to support RHEL5 as a baseline, which is 10 years old by now, I'm sort of uncertain.

FWIW, I don't know if this affects RHEL5 too. Comment 25 indicates that this might have regressed from 6.6 to 6.7, but that could have been some happenstance change in load order. Also, RHEL5 is in the extended life phase, so I doubt this would be considered severe enough for an update.

I think it might be a good idea to have it on by default, and then have a built-time option for very out-of-date operating systems to disable it in that case.

You'd have to disable it on rustc/etc. for those systems to run upstream compiler binaries at all.

I think relro alone is OK, and perhaps -z,relro and -z,now could both get nice -C options to enable them, rather than having to use raw -Clink-arg.

cuviper · 2017-07-14T18:10:36Z

FWIW, I don't know if this affects RHEL5 too.

Actually, rust's dist-powerpc64-linux is only configured for ~RHEL6, so it would have problems with glibc symbol versions if you tried to run on RHEL5-ppc64 at all.

I think only the x86 builders are older, and they're running on native CentOS 5 images, so they would probably fail directly while bootstrapping if that glibc had BIND_NOW problems.

alexcrichton · 2017-07-14T19:17:20Z

If this ends up being an powerpc64-specific bug then I think it's also fine to just disable it there with a comment indicating why.

It also seems like the risk of breakage from this is pretty low, so we could always land it and see what happens? Backing out the change would be pretty easy as well, so I think it may not cause too much of a splash to implement.

If we end up having problems during the x86 bootstrap within CentOS 5 the bors will discover that anyway as this won't be able to land!

@cuviper what do you think of the "land and let's see" approach? Should we disable full relro on powerpc64 for now but leave partial relro enabled?

I think regardless we'll hold off on this for a few days to land this after the beta branching on this coming Monday just to make sure we have a maximal amount of time to detect regressions.

cuviper · 2017-07-14T19:57:06Z

I'm not positive that bootstrap will exercise this in the CentOS 5 builders. In stage0 we'll build a stage1 without relro, then stage1 will build stage2 with relro. But then I think we don't actually execute stage2 for anything unless you run tests, right? (Even just rustc -V will do...)

Anyway, if we want to "land and let's see", I can make a point of trying the first relro nightly on the various arches that Red Hat and Fedora support. I would prefer we avoid BIND_NOW on powerpc64 though, since we already know that's problematic.

Signed-off-by: Johannes Löthberg <[email protected]>

kyrias · 2017-07-14T20:09:52Z

Hokay, so I've now:

Added support for both full and partial RELRO
Added a -C flag called relro-level that lets you select between full, partial, or off
Made ppc64 targets use partial RELRO by default rather than full.

cuviper · 2017-07-14T20:14:06Z

src/librustc_back/target/powerpc64le_unknown_linux_gnu.rs


 pub fn target() -> TargetResult {
    let mut base = super::linux_base::opts();
    base.cpu = "ppc64le".to_string();
    base.pre_link_args.get_mut(&LinkerFlavor::Gcc).unwrap().push("-m64".to_string());
    base.max_atomic_width = Some(64);

+    // ld.so in at least RHEL6 on ppc64 has a bug related to BIND_NOW, so only enable partial RELRO
+    // for now. https://github.com/rust-lang/rust/pull/43170#issuecomment-315411474
+    base.relro_level = RelroLevel::Partial;


ppc64le didn't exist until RHEL7, so I think we can leave that full.

Ah I see, will undo that part, thanks.

On at least RHEL6 there is a segfault caused by the older ld.so version when BIND_NOW is used, so use partial RELRO by default on ppc64 architectures for now. Signed-off-by: Johannes Löthberg <[email protected]>

Signed-off-by: Johannes Löthberg <[email protected]>

alexcrichton · 2017-07-18T15:56:31Z

@bors: r+

Thanks @kyrias!

bors · 2017-07-18T15:56:32Z

📌 Commit 2161fb2 has been approved by alexcrichton

bors · 2017-07-19T05:39:19Z

⌛ Testing commit 2161fb2 with merge 4e56bbe...

Add support for full RELRO This commit adds support for full RELRO, and enables it for the platforms I know have support for it. Full RELRO makes the PLT+GOT data read-only on startup, preventing it from being overwritten. http://tk-blog.blogspot.com/2009/02/relro-not-so-well-known-memory.html Fixes #29877. --- I'm not entirely certain if this is the best way to do it, but I figured mimicking the way it's done for PIE seemed like a good start at least. I'm not sure whether we want to have it enabled by default globally and then disabling it explicitly for targets that don't support it though. I'm also not sure whether the `full_relro` function should call `bug!()` or something like it for linkers that don't support it rather than no-opping.

bors · 2017-07-19T08:33:02Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 4e56bbe to master...

lilianmoraru · 2017-08-19T10:29:22Z

@kyrias I try to use this and it doesn't seem to work for me.
RUSTFLAGS="-C relro-level=full" cargo +nightly build
or
rustc +nightly -C relro-level=help

rustc version: rustc 1.21.0-nightly (df511d554 2017-08-14)
Am I using the flags incorrectly?

kyrias · 2017-08-19T14:18:46Z

@lilianmoraru It's currently under -Z to mark it as an unstable flag.

Add relro-level tests The `relro-level` debugging flag was added in #43170 which was merged in July 2017. This PR moves this flag to be a proper codegen flag.

@michaelwoerister

[WIP] Support for disabling PLT for better function call performance This PR gives `rustc` the ability to skip the PLT when generating function calls into shared libraries. This can improve performance by reducing branch indirection. AFAIK, the only advantage of using the PLT is to allow for ELF lazy binding. However, since Rust already [enables full relro for security](#43170), lazy binding was disabled anyway. This is a little known feature which is supported by [GCC](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html) and [Clang](https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-fplt) as `-fno-plt` (some Linux distros [enable it by default](https://git.archlinux.org/svntogit/packages.git/tree/trunk/makepkg.conf?h=packages/pacman#n40) for all builds). Implementation inspired by [this patch](https://reviews.llvm.org/D39079#change-YvkpNDlMs_LT) which adds `-fno-plt` support to Clang. ## Performance I didn't run a lot of benchmarks, but these are the results on my machine for a `clap` [benchmark](https://github.com/clap-rs/clap/blob/master/benches/05_ripgrep.rs): ``` name control ns/iter no-plt ns/iter diff ns/iter diff % speedup build_app_long 11,097 10,733 -364 -3.28% x 1.03 build_app_short 11,089 10,742 -347 -3.13% x 1.03 build_help_long 186,835 182,713 -4,122 -2.21% x 1.02 build_help_short 80,949 78,455 -2,494 -3.08% x 1.03 parse_clean 12,385 12,044 -341 -2.75% x 1.03 parse_complex 19,438 19,017 -421 -2.17% x 1.02 parse_lots 431,493 421,421 -10,072 -2.33% x 1.02 ``` A small performance improvement across the board, with no downsides. It's likely binaries which make a lot of function calls into dynamic libraries could see even more improvements. [This comment](https://patchwork.ozlabs.org/patch/468993/#1028255) suggests that, in some cases, `-fno-plt` could improve PIC/PIE code performance by 10%. ## To do - [ ] Do a perf run to see the effect this has on the compiler (cc @michaelwoerister), and possibly run benchmarks on some more crates - [ ] Add a code gen test - [ ] Should this be always enabled or should it be behind a command line option? If so, what should it be called? `-Z no-plt`? `-Z plt=no`?

Support for disabling PLT for better function call performance This PR gives `rustc` the ability to skip the PLT when generating function calls into shared libraries. This can improve performance by reducing branch indirection. AFAIK, the only advantage of using the PLT is to allow for ELF lazy binding. However, since Rust already [enables full relro for security](#43170), lazy binding was disabled anyway. This is a little known feature which is supported by [GCC](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html) and [Clang](https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-fplt) as `-fno-plt` (some Linux distros [enable it by default](https://git.archlinux.org/svntogit/packages.git/tree/trunk/makepkg.conf?h=packages/pacman#n40) for all builds). Implementation inspired by [this patch](https://reviews.llvm.org/D39079#change-YvkpNDlMs_LT) which adds `-fno-plt` support to Clang. ## Performance I didn't run a lot of benchmarks, but these are the results on my machine for a `clap` [benchmark](https://github.com/clap-rs/clap/blob/master/benches/05_ripgrep.rs): ``` name control ns/iter no-plt ns/iter diff ns/iter diff % speedup build_app_long 11,097 10,733 -364 -3.28% x 1.03 build_app_short 11,089 10,742 -347 -3.13% x 1.03 build_help_long 186,835 182,713 -4,122 -2.21% x 1.02 build_help_short 80,949 78,455 -2,494 -3.08% x 1.03 parse_clean 12,385 12,044 -341 -2.75% x 1.03 parse_complex 19,438 19,017 -421 -2.17% x 1.02 parse_lots 431,493 421,421 -10,072 -2.33% x 1.02 ``` A small performance improvement across the board, with no downsides. It's likely binaries which make a lot of function calls into dynamic libraries could see even more improvements. [This comment](https://patchwork.ozlabs.org/patch/468993/#1028255) suggests that, in some cases, `-fno-plt` could improve PIC/PIE code performance by 10%. ## Security benefits **Bonus**: some of the speculative execution attacks rely on the PLT, by disabling it we reduce a big attack surface and reduce the need for [`retpoline`](https://reviews.llvm.org/D41723). ## Remaining PLT calls The compiled binaries still have plenty of PLT calls, coming from C/C++ libraries. Building dependencies with `CFLAGS=-fno-plt CXXFLAGS=-fno-plt` removes them.

rust-highfive assigned eddyb Jul 11, 2017

rust-highfive assigned alexcrichton and unassigned eddyb Jul 11, 2017

arielb1 added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 11, 2017

Support both partial and full RELRO

94b9cc9

Signed-off-by: Johannes Löthberg <[email protected]>

cuviper reviewed Jul 14, 2017

View reviewed changes

Make partial RELRO default on ppc64 due to segfault

ecf3f6d

On at least RHEL6 there is a segfault caused by the older ld.so version when BIND_NOW is used, so use partial RELRO by default on ppc64 architectures for now. Signed-off-by: Johannes Löthberg <[email protected]>

kyrias added 2 commits July 18, 2017 00:18

Move relro_level from CodegenOptions to DebuggingOptions

6a8328c

Signed-off-by: Johannes Löthberg <[email protected]>

Implement FromStr for RelroLevel rather than duplicating the match

2161fb2

Signed-off-by: Johannes Löthberg <[email protected]>

arielb1 added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 18, 2017

bors merged commit 2161fb2 into rust-lang:master Jul 19, 2017

bors mentioned this pull request Jul 19, 2017

Add empty MIR pass for non-lexical lifetimes #43271

Merged

kyrias deleted the full-relro branch July 20, 2017 00:14

alexcrichton added the relnotes Marks issues that should be documented in the release notes of the next release. label Jul 20, 2017

Mark-Simulacrum mentioned this pull request Jul 26, 2017

Nightlies since 2017-07-20 break when linking some libraries targeting Musl #43486

Closed

kyrias mentioned this pull request Feb 20, 2018

Add relro-level tests #48388

Merged

GabrielMajeri mentioned this pull request Sep 26, 2018

Support for disabling PLT for better function call performance #54592

Merged

Blacksmoke16 mentioned this pull request Jul 31, 2021

Support for Full RELRO crystal-lang/crystal#11046

Open

davidtwco mentioned this pull request Feb 27, 2024

sess: stabilize -Zrelro-level as -Crelro-level #121694

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for full RELRO #43170

Add support for full RELRO #43170

kyrias commented Jul 11, 2017

rust-highfive commented Jul 11, 2017

eddyb commented Jul 11, 2017

cuviper commented Jul 11, 2017

alexcrichton commented Jul 11, 2017

cuviper commented Jul 11, 2017

kyrias commented Jul 12, 2017

alexcrichton commented Jul 12, 2017

parched commented Jul 12, 2017

kyrias commented Jul 12, 2017

kyrias commented Jul 12, 2017

cuviper commented Jul 12, 2017

cuviper commented Jul 13, 2017

alexcrichton commented Jul 13, 2017

cuviper commented Jul 14, 2017

kyrias commented Jul 14, 2017

cuviper commented Jul 14, 2017

kyrias commented Jul 14, 2017

cuviper commented Jul 14, 2017

cuviper commented Jul 14, 2017

alexcrichton commented Jul 14, 2017

cuviper commented Jul 14, 2017

kyrias commented Jul 14, 2017

cuviper Jul 14, 2017

kyrias Jul 14, 2017

alexcrichton commented Jul 18, 2017

bors commented Jul 18, 2017

bors commented Jul 19, 2017

bors commented Jul 19, 2017

lilianmoraru commented Aug 19, 2017

kyrias commented Aug 19, 2017

Add support for full RELRO #43170

Add support for full RELRO #43170

Conversation

kyrias commented Jul 11, 2017

rust-highfive commented Jul 11, 2017

eddyb commented Jul 11, 2017

cuviper commented Jul 11, 2017

alexcrichton commented Jul 11, 2017

cuviper commented Jul 11, 2017

kyrias commented Jul 12, 2017

alexcrichton commented Jul 12, 2017

parched commented Jul 12, 2017

kyrias commented Jul 12, 2017

kyrias commented Jul 12, 2017

cuviper commented Jul 12, 2017

cuviper commented Jul 13, 2017

alexcrichton commented Jul 13, 2017

cuviper commented Jul 14, 2017

kyrias commented Jul 14, 2017

cuviper commented Jul 14, 2017

kyrias commented Jul 14, 2017

cuviper commented Jul 14, 2017

cuviper commented Jul 14, 2017

alexcrichton commented Jul 14, 2017

cuviper commented Jul 14, 2017

kyrias commented Jul 14, 2017

cuviper Jul 14, 2017

Choose a reason for hiding this comment

kyrias Jul 14, 2017

Choose a reason for hiding this comment

alexcrichton commented Jul 18, 2017

bors commented Jul 18, 2017

bors commented Jul 19, 2017

bors commented Jul 19, 2017

lilianmoraru commented Aug 19, 2017

kyrias commented Aug 19, 2017