complicated build stages

[onur-ozkan]

Current approach on bootstrap stages adds too much complexity on the bootstrap codebase; the complexity continues to grow with the additional changes made on bootstrap and it's not even functioning correctly (like writing incorrect stages to stdout on certain commands).

One possible solution could be to handle stage behaviours as part of the Step trait individually on Step implementation.

Some references to why current implementation of step calculations is making things so difficult/complicated:

rust/src/bootstrap/src/core/build_steps/compile.rs

Lines 839 to 847 in ef32456

	impl Step for Rustc {
	// We return the stage of the "actual" compiler (not the uplifted one).
	//
	// By "actual" we refer to the uplifting logic where we may not compile the requested stage;
	// instead, we uplift it from the previous stages. Which can lead to bootstrap failures in
	// specific situations where we request stage X from other steps. However we may end up
	// uplifting it from stage Y, causing the other stage to fail when attempting to link with
	// stage X which was never actually built.
	type Output = u32;

rust/src/bootstrap/src/core/build_steps/compile.rs

Lines 905 to 926 in ef32456

	let compiler_to_use = builder.compiler_for(compiler.stage, compiler.host, target);
	if compiler_to_use != compiler {
	builder.ensure(Rustc::new(compiler_to_use, target));
	let msg = if compiler_to_use.host == target {
	format!(
	"Uplifting rustc (stage{} -> stage{})",
	compiler_to_use.stage,
	compiler.stage + 1
	)
	} else {
	format!(
	"Uplifting rustc (stage{}:{} -> stage{}:{})",
	compiler_to_use.stage,
	compiler_to_use.host,
	compiler.stage + 1,
	target
	)
	};
	builder.info(&msg);
	builder.ensure(RustcLink::from_rustc(self, compiler_to_use));
	return compiler_to_use.stage;
	}

Particularly, the behaviors of the force_use_stage function are not really useful for all use cases (for instance, see #118233 and #118918)

rust/src/bootstrap/src/lib.rs

Lines 1403 to 1417 in ef32456

	fn force_use_stage1(&self, stage: u32, target: TargetSelection) -> bool {
	!self.config.full_bootstrap
	&& !self.config.download_rustc()
	&& stage >= 2
	&& (self.hosts.iter().any(|h| *h == target) || target == self.build)
	}
	/// Checks whether the `compiler` compiling for `target` should be forced to
	/// use a stage2 compiler instead.
	///
	/// When we download the pre-compiled version of rustc and compiler stage is >= 2,
	/// it should be forced to use a stage2 compiler.
	fn force_use_stage2(&self, stage: u32) -> bool {
	self.config.download_rustc() && stage >= 2
	}

@rustbot label +T-bootstrap

[RalfJung]

Note that there's been lots of prior discussion on this:

• x.py's naming of stages is confusing #59864
• Explain stages (take N) rustc-dev-guide#843
• Explain stages in terms of the compiler currently running (take N+1) rustc-dev-guide#857
• Redesign bootstrap stages compiler-team#619

[RalfJung]

I think some of this logic is also just wrong. For instance:

rust/src/bootstrap/src/core/build_steps/tool.rs

Lines 822 to 840 in 288daeb

	if (false $(|| !$add_bins_to_sysroot.is_empty())?) && $sel.compiler.stage > 0 {
	let bindir = $builder.sysroot($sel.compiler).join("bin");
	t!(fs::create_dir_all(&bindir));
	#[allow(unused_variables)]
	let tools_out = $builder
	.cargo_out($sel.compiler, Mode::ToolRustc, $sel.target);
	$(for add_bin in $add_bins_to_sysroot {
	let bin_source = tools_out.join(exe(add_bin, $sel.target));
	let bin_destination = bindir.join(exe(add_bin, $sel.compiler.host));
	$builder.copy_link(&bin_source, &bin_destination);
	})?
	let tool = bindir.join(exe($tool_name, $sel.compiler.host));
	tool
	} else {
	tool
	}

This copies the clippy/miri built with the stage N compiler into the stage N sysroot. But that makes no sense: for rustc we copy the rustc built with the stage N compiler into the stage N+1 sysroot. The same should happen here.

Currently, miri.exe that is copied to the stage 1 sysroot needs to be loaded with the libraries that are in the stage 2 sysroot. Clearly that makes no sense.

[RalfJung]

So specifically, currently we're copying from stage1-tools-bin to stage1/bin. But we're also copying from stage1-rustc to stage2/bin. One of these has to be wrong, I think -- and I think it's the tools that are wrong.

[onur-ozkan]

Some references to why current implementation of step calculations is making things so difficult/complicated:

Additional ref. to this issue: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Stage.20for.20copying.20tools.20into.20the.20sysroot