Make all alt builders produce parallel-enabled compilers #64722

Mark-Simulacrum · 2019-09-23T23:32:03Z

We're not quite ready to ship parallel compilers by default, but the alt
builders are not used too much (in theory), so we believe that shipping
a possibly-broken compiler there is not too problematic.

r? @nikomatsakis

We're not quite ready to ship parallel compilers by default, but the alt builders are not used too much (in theory), so we believe that shipping a possibly-broken compiler there is not too problematic.

Mark-Simulacrum · 2019-09-23T23:34:30Z

Going to nominate for infra and compiler teams as I'd like to check that the claims I've made are true:

usage is minimal, mostly to deal with LLVM-related regressions (-Zthreads=1 is probably enough to sidestep any bugs in parallel that could impact this)
we are comfortable altering them in such a "big" way.

pnkfelix · 2019-09-26T14:55:18Z

Discussed in T-compiler meeting. We ourselves don't see any problem with doing this as described here.

In particular, @nikomatsakis wanted to stress that we do plan to keep LLVM assertions enabled for these builds. This will mean that these builds help flush out parallel bugs, but they will not be a good basis for comparative benchmarking.

pnkfelix · 2019-09-26T14:55:51Z

(leaving nominated tag on, but at this point that is directed at T-infra, not T-compiler.)

Mark-Simulacrum · 2019-09-30T19:00:58Z

r? @alexcrichton

I believe we discussed this in the infra meeting and there weren't any major objections -- could you just make sure this is doing what it's intended and such and approve if you think we're good to go here?

alexcrichton · 2019-09-30T19:10:09Z

Would it be possible to configure these compilers to be built with -j1 for everything except the codegen backend? I'm basically curious if we can just enable parallelism but not use it.

Mark-Simulacrum · 2019-09-30T19:16:16Z

I think that's "just" -Zthreads=1 at runtime but I can take a look to both verify that theory and, if so, hard code that as the default (perhaps via a cfg flag or something).

alexcrichton · 2019-09-30T19:45:45Z

Ah ok, great! In that case actually, I think it'd be best to default to -Zthreads=1 by default? Basically leave everything as-is, but just producing "parallel capable" compilers here.

Mark-Simulacrum · 2019-09-30T20:30:52Z

Alright, I pushed up another commit that makes the default for query parallelism 1, instead of num_cpus as before. This does not affect the codegen parallelism.

alexcrichton · 2019-09-30T20:37:25Z

src/librustc/session/config.rs

@@ -1259,7 +1259,7 @@ options! {DebuggingOptions, DebuggingSetter, basic_debugging_options,
        "prints the LLVM optimization passes being run"),
    ast_json: bool = (false, parse_bool, [UNTRACKED],
        "print the AST as JSON and halt"),
-    threads: Option<usize> = (None, parse_opt_uint, [UNTRACKED],
+    threads: Option<usize> = (Some(1), parse_opt_uint, [UNTRACKED],


It seems a bit odd to have this be Option<usize> when the default isn't None, could this instead change threads_from_count, and a comment could be included why it's defaulted to one?

Sure, yeah.

This changes the default parallelism for parallel compilers to one, instead of the previous default, which was "num cpus". This is likely not an optimal default long-term, but it is a good default for testing whether parallel compilers are not a significant regression over a sequential compiler. Notably, this in theory makes a parallel-enabled compiler behave exactly like a sequential compiler with respect to the jobserver.

alexcrichton · 2019-09-30T20:50:45Z

@bors: r+

bors · 2019-09-30T20:50:46Z

📌 Commit f271c155f02c8a5fd7e079d1d7f21d188ee1f8f9 has been approved by alexcrichton

Mark-Simulacrum · 2019-09-30T20:51:43Z

Was a bit overeager pushing, there's a new commit up (no significant differences)

alexcrichton · 2019-09-30T21:09:09Z

@bors: r+

bors · 2019-09-30T21:09:10Z

📌 Commit 1a1067d has been approved by alexcrichton

@nikomatsakis

…xcrichton Make all alt builders produce parallel-enabled compilers We're not quite ready to ship parallel compilers by default, but the alt builders are not used too much (in theory), so we believe that shipping a possibly-broken compiler there is not too problematic. r? @nikomatsakis

@nikomatsakis

…xcrichton Make all alt builders produce parallel-enabled compilers We're not quite ready to ship parallel compilers by default, but the alt builders are not used too much (in theory), so we believe that shipping a possibly-broken compiler there is not too problematic. r? @nikomatsakis

@nikomatsakis

…xcrichton Make all alt builders produce parallel-enabled compilers We're not quite ready to ship parallel compilers by default, but the alt builders are not used too much (in theory), so we believe that shipping a possibly-broken compiler there is not too problematic. r? @nikomatsakis

@nikomatsakis

…xcrichton Make all alt builders produce parallel-enabled compilers We're not quite ready to ship parallel compilers by default, but the alt builders are not used too much (in theory), so we believe that shipping a possibly-broken compiler there is not too problematic. r? @nikomatsakis

@ghost

Rollup of 11 pull requests Successful merges: - #64649 (Avoid ICE on return outside of fn with literal array) - #64722 (Make all alt builders produce parallel-enabled compilers) - #64801 (Avoid `chain()` in `find_constraint_paths_between_regions()`.) - #64805 (Still more `ObligationForest` improvements.) - #64840 (SelfProfiler API refactoring and part one of event review) - #64885 (use try_fold instead of try_for_each to reduce compile time) - #64942 (Fix clippy warnings) - #64952 (Update cargo.) - #64974 (Fix zebra-striping in generic dataflow visualization) - #64978 (Fully clear `HandlerInner` in `Handler::reset_err_count`) - #64979 (Update books) Failed merges: - #64959 (syntax: improve parameter without type suggestions) r? @ghost

Make all alt builders produce parallel-enabled compilers

a3639c6

We're not quite ready to ship parallel compilers by default, but the alt builders are not used too much (in theory), so we believe that shipping a possibly-broken compiler there is not too problematic.

Mark-Simulacrum added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Sep 23, 2019

rust-highfive assigned nikomatsakis Sep 23, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 23, 2019

rust-highfive assigned alexcrichton and unassigned nikomatsakis Sep 30, 2019

alexcrichton reviewed Sep 30, 2019

View reviewed changes

Mark-Simulacrum force-pushed the alt-parallel branch from 550df4d to f271c15 Compare September 30, 2019 20:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). labels Sep 30, 2019

Mark-Simulacrum force-pushed the alt-parallel branch from f271c15 to 1a1067d Compare September 30, 2019 20:51

Mark-Simulacrum added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed I-nominated S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 30, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 30, 2019

Centril mentioned this pull request Oct 1, 2019

Rollup of 5 pull requests #64955

Closed

tmandry mentioned this pull request Oct 1, 2019

Rollup of 7 pull requests #64965

Closed

tmandry mentioned this pull request Oct 1, 2019

Rollup of 8 pull requests #64966

Closed

tmandry mentioned this pull request Oct 2, 2019

Rollup of 11 pull requests #64981

Merged

bors merged commit 1a1067d into rust-lang:master Oct 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make all alt builders produce parallel-enabled compilers #64722

Make all alt builders produce parallel-enabled compilers #64722

Mark-Simulacrum commented Sep 23, 2019

Mark-Simulacrum commented Sep 23, 2019

pnkfelix commented Sep 26, 2019

pnkfelix commented Sep 26, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton commented Sep 30, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton commented Sep 30, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton Sep 30, 2019

Mark-Simulacrum Sep 30, 2019

alexcrichton commented Sep 30, 2019

bors commented Sep 30, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton commented Sep 30, 2019

bors commented Sep 30, 2019

Make all alt builders produce parallel-enabled compilers #64722

Make all alt builders produce parallel-enabled compilers #64722

Conversation

Mark-Simulacrum commented Sep 23, 2019

Mark-Simulacrum commented Sep 23, 2019

pnkfelix commented Sep 26, 2019

pnkfelix commented Sep 26, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton commented Sep 30, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton commented Sep 30, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton Sep 30, 2019

Choose a reason for hiding this comment

Mark-Simulacrum Sep 30, 2019

Choose a reason for hiding this comment

alexcrichton commented Sep 30, 2019

bors commented Sep 30, 2019

Mark-Simulacrum commented Sep 30, 2019

alexcrichton commented Sep 30, 2019

bors commented Sep 30, 2019