Running ./x.py test miri --stage 0 twice rebuilds miri, cargo-miri, and rustdoc

[RalfJung]

This is fairly easy to reproduce:

./x.py test miri --stage 0 --test-args hello
./x.py test miri --stage 0 --test-args hello

(The --test-args just serves to make this take less long.)

The second command shouldn't have to build anything, just run the tests. But somehow it actually rebuilds miri, cargo-miri, and rustdoc.

This is a recent regression introduced by #123028. Somehow building rustdoc seems to destroy the build cache for miri and cargo-miri, and vice versa.
Cc @onur-ozkan

[onur-ozkan]

Building rustdoc using the stage 1 compiler causes this, which is weird. I suspect that previously invoked miri related steps may have broken the cache somehow. Or we may need to improve/enhance Builder::ensure.

[RalfJung]

I've just found this in the rustdoc sources:

rust/src/bootstrap/src/core/build_steps/tool.rs

Lines 441 to 445 in 7bdd63d

	// Similar to `compile::Assemble`, build with the previous stage's compiler. Otherwise
	// we'd have stageN/bin/rustc and stageN/bin/rustdoc be effectively different stage
	// compilers, which isn't what we want. Rustdoc should be linked in the same way as the
	// rustc compiler it's paired with, so it must be built with the previous stage compiler.
	let build_compiler = builder.compiler(target_compiler.stage - 1, builder.config.build);

Maybe Miri should do something similar. Then ./x.py test miri should also Do The Right Thing without adding --stage 0 (since actually stage 1 would be the right stage to use). And maybe then the rebuilding issue also vanishes. Or maybe that's unrelated, not sure.

[onur-ozkan]

That would work actually, it seems to be the right path.

[RalfJung]

Unfortunately it doesn't help. We probably still should do it.

But somehow building rustdoc seems to delete the build cache for Miri and cargo-miri, and vice versa.

[RalfJung]

Cargo has this to say about why things are rebuilt:

       Dirty libc v0.2.153: the rustflags changed

And indeed, rustdoc sets

cargo.rustflag("--cfg=parallel_compiler");

That's a bit annoying to have this set in the tool, it means each tool needs to repeat the same logic.

[onur-ozkan]

iirc rustdoc sets that to not dirt the compiler builds, we should probably handle that more globally.

[RalfJung]

iirc rustdoc sets that to not dirt the compiler builds, we should probably handle that more globally.

Yup, that's what the last commit in #123192 does now.

[RalfJung]

This issue is back now, I am again seeing rebuilds of Miri and rustdoc all the time. Did something change in bootstrap?

[onur-ozkan]

Couldn't reproduce it on my side with default config (without the config.toml file).

$ ./x build rustdoc
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.03s
WARNING: you have not made a `config.toml`
HELP: consider running `./x.py setup` or copying `config.example.toml` by running `cp config.example.toml config.toml`
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.10s
Building compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.17s
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
Building stage0 tool lld-wrapper (x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.09s
Building tool rustdoc (stage0 -> stage1, x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.11s
WARNING: you have not made a `config.toml`
HELP: consider running `./x.py setup` or copying `config.example.toml` by running `cp config.example.toml config.toml`
NOTE: this message was printed twice to make it more likely to be seen
Build completed successfully in 0:00:02

$ git log -1
commit 164e1297e1bce47a241e4d93a9f044452b288715 (HEAD -> master, origin/master, origin/HEAD)
Merge: fc555cd832e 45da03541c0
Author: bors <[email protected]>
Date:   Tue Jun 25 05:09:30 2024 +0000

    Auto merge of #125610 - oli-obk:define_opaque_types14, r=compiler-errors

    Allow constraining opaque types during various unsizing casts

    allows unsizing of tuples, arrays and Adts to constraint opaque types in their generic parameters to concrete types on either side of the unsizing cast.

    Also allows constraining opaque types during trait object casts that only differ in auto traits or lifetimes.

    cc #116652

There might be a specific option that triggers this. Can you share your config?

[onur-ozkan]

Couldn't reproduce it on my side with default config (without the config.toml file).

$ ./x build rustdoc
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.03s
WARNING: you have not made a `config.toml`
HELP: consider running `./x.py setup` or copying `config.example.toml` by running `cp config.example.toml config.toml`
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.10s
Building compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.17s
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
Building stage0 tool lld-wrapper (x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.09s
Building tool rustdoc (stage0 -> stage1, x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.11s
WARNING: you have not made a `config.toml`
HELP: consider running `./x.py setup` or copying `config.example.toml` by running `cp config.example.toml config.toml`
NOTE: this message was printed twice to make it more likely to be seen
Build completed successfully in 0:00:02

$ git log -1
commit 164e1297e1bce47a241e4d93a9f044452b288715 (HEAD -> master, origin/master, origin/HEAD)
Merge: fc555cd832e 45da03541c0
Author: bors <[email protected]>
Date:   Tue Jun 25 05:09:30 2024 +0000

    Auto merge of #125610 - oli-obk:define_opaque_types14, r=compiler-errors

    Allow constraining opaque types during various unsizing casts

    allows unsizing of tuples, arrays and Adts to constraint opaque types in their generic parameters to concrete types on either side of the unsizing cast.

    Also allows constraining opaque types during trait object casts that only differ in auto traits or lifetimes.

    cc #116652

There might be a specific option that triggers this. Can you share your config?

Seems like it only happens with x test caused by different rustflags.

[RalfJung]

The command I was running is ./x.py miri library/core --test-args notest. Running that command multiple times in a row rebuilds miri and rustdoc each time. Adding --no-doc avoids that.

[onur-ozkan]

Found the problem and opened the fix PR. I will try to add a bootstrap test (around this weekend) for ensuring this never happens again.

[RalfJung]

Thanks for the quick debugging :)