Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent extensions from blocking parallel pre-compilation #55910

Merged
merged 3 commits into from
Nov 21, 2024

Conversation

topolarity
Copy link
Member

@topolarity topolarity commented Sep 27, 2024

Previously our precompilation code was causing anything with package A as a dependency to wait on all of A's extensions and weakdeps to finish before starting to pre-compile, even if it can't actually load those weakdeps (or the extensions themselves)

This would lead to a pre-compile ordering like:

A        B
 \      / \
  Ext AB   \
     |     /
     C    /
      \  /
       D

Here C cannot pre-compile in parallel with Ext {A,B} and B, because it has to wait for Ext {A,B} to finish pre-compiling. That happens even though C has no way to load either of these.

This change updates the pre-compile ordering to be more parallel, reflecting the true place where Ext {A,B} can be loaded:

  A       B
 / \     / \
C   Ext AB  |
 \    |    /
  \-- D --/

which allows C to compile in parallel with B and Ext{A,B}

Please review 55b40ed (that's the only commit w/ important changes)

@topolarity topolarity added backport 1.10 Change should be backported to the 1.10 release backport 1.11 Change should be backported to release-1.11 labels Sep 27, 2024
@KristofferC KristofferC mentioned this pull request Sep 30, 2024
39 tasks
@topolarity topolarity force-pushed the ct/fix-pkg-ext-deps branch 2 times, most recently from fcdd78b to 55b40ed Compare September 30, 2024 13:20
KristofferC added a commit that referenced this pull request Oct 1, 2024
Backported PRs:
- [x] #55849 <!-- Mmap: fix grow! for non file IOs -->
- [x] #55863 <!-- Update TaskLocalRNG docstring according to #49110 -->
- [x] #54433 <!-- Root globals in toplevel exprs -->
- [x] #55828 <!-- Fix some corner cases of `isapprox` with unsigned
integers -->
- [x] #55890 <!-- Profile: fix order of fields in heapsnapshot & improve
formatting -->
- [x] #55884 <!-- inference: add missing `TypeVar` handling for
`instanceof_tfunc` -->
- [x] #55881 <!-- Install terminfo data under /usr/share/julia -->
- [x] #55909 <!-- do not intentionally suppress errors in precompile
script from being reported or failing the result -->
- [x] #55355 <!-- relocation: account for trailing path separator in
depot paths -->
- [x] #55917 <!-- fix rawbigints OOB issues -->
- [x] #55892 <!-- TOML: Avoid type-pirating `Base.TOML.Parser` -->
- [x] #55798 <!-- Broadcast binary ops involving strided triangular -->
- [x] #55919 <!-- Limit `@inbounds` to indexing in the dual-iterator
branch in `copyto_unaliased!` -->

Contains multiple commits, manual intervention needed:
- [ ] #54009 <!-- allow extensions to trigger from packages in [deps]
-->
- [ ] #55509 <!-- Fix cong implementation to be properly random and not
just cycling. -->
- [ ] #55569 <!-- Add a docs section about loading/precomp/ttfx time
tuning -->
- [ ] #55824 <!-- Replace regex package module checks with actual code
checks -->

Non-merged PRs with backport label:
- [ ] #55932 <!-- REPL: make UndefVarError aware of imported modules -->
- [ ] #55910 <!-- Prevent extensions from blocking parallel
pre-compilation -->
- [ ] #55908 <!-- add logic to prefer loading modules that are already
loaded -->
- [ ] #55886 <!-- irrationals: restrict assume effects annotations to
known types -->
- [ ] #55871 <!-- lowering: don't reverse handler order in
`(pop-handler-list ...)` -->
- [ ] #55870 <!-- fix infinite recursion in `promote_type` for
`Irrational` -->
- [ ] #55867 <!-- update `hash` doc string: `widen` not required any
more -->
- [ ] #55851 <!-- [REPL] Fix #55850 by using `safe_realpath` instead of
`abspath` in `projname` -->
- [ ] #55813 <!-- Check for conflicting `@ccallable` name before JIT
registration -->
- [ ] #55743 <!-- doc: heap snapshot viewing -->
- [ ] #55741 <!-- Change annotations to use a NamedTuple -->
- [ ] #55534 <!-- Set stdlib sources as read-only during installation
-->
- [ ] #55499 <!-- propagate the terminal's `displaysize` to the
`IOContext` used by the REPL -->
- [ ] #55458 <!-- Allow for generically extracting unannotated string
-->
- [ ] #55457 <!-- Make AnnotateChar equality consider annotations -->
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
This was referenced Oct 7, 2024
@KristofferC KristofferC mentioned this pull request Oct 18, 2024
43 tasks
@KristofferC KristofferC mentioned this pull request Oct 29, 2024
47 tasks
@KristofferC
Copy link
Member

Bump

@topolarity
Copy link
Member Author

Bump? This is waiting for review

base/precompilation.jl Outdated Show resolved Hide resolved
@vtjnash
Copy link
Member

vtjnash commented Nov 18, 2024

Is that counting as a review?

@KristofferC
Copy link
Member

yes

@topolarity
Copy link
Member Author

This change is needed for correct-ness, as it turns out.

The existing behavior where we lift to just under the parent tightens the scheduling constraints on triggers too much, leading to false cycles. For example ExtBA and A depends on B starts as:

A -> B

but after injection of ExtBA under its parent B this becomes:

A -> ExtBA -> B
 ^-----/

which has put A and ExtBA into a false cycle. The correct scheduling here is obviously:

ExtBA -> A -> B

Basically the way that ExtBA was added to this graph did not realize that one of the triggers (A) is constrained to compile after the parent already. If we wanted to avoid the cycle, we needed to attach to B instead (or to the LCA, as this PR does)

@IanButterworth
Copy link
Member

Makes sense to me from the description.

topolarity added a commit to topolarity/julia that referenced this pull request Nov 20, 2024
This change is a slimmed-down version of JuliaLang#55910, without any of the
re-factoring.

This improves the parallelism of the pre-compile job and fixes a
correctness bug, where unconditionally injecting a dependency after its
parent could be in conflict with dependencies in between triggers,
causing false extension cycles.

For example adding `AMDGPU.jl` and `Tracker.jl` in your project reports
a false cycle (without this change).
topolarity added a commit to topolarity/julia that referenced this pull request Nov 20, 2024
This change is a slimmed-down version of JuliaLang#55910, without any of the
re-factoring.

This improves the parallelism of the pre-compile job and fixes a
correctness bug, where unconditionally injecting a dependency after its
parent could be in conflict with dependencies in between triggers,
causing false extension cycles.

For example adding `AMDGPU.jl` and `Tracker.jl` in your project reports
a false cycle (without this change).
topolarity added a commit to topolarity/julia that referenced this pull request Nov 20, 2024
This change is a slimmed-down version of JuliaLang#55910, without any of the
re-factoring.

This improves the parallelism of the pre-compile job and fixes a
correctness bug, where unconditionally injecting a dependency after its
parent could be in conflict with dependencies in between triggers,
causing false extension cycles.

For example adding `AMDGPU.jl` and `Tracker.jl` in your project reports
a false cycle (without this change).
Previously our precompilation code was causing any dependencies of a
package A to wait on all of A's weakdeps to finish pre-compiling,
even if it can't actually load those weakdeps (or the extension itself)

This would lead to a pre-compile ordering like:
```
A        B
 \      / \
  Ext AB   \
     |     /
     C    /
      \  /
       D
```

Here, extension `C` cannot pre-compile in parallel with `Ext {A,B}` and
`B`, because it has to wait for `Ext {A,B}` to finish pre-compiling.
That happens even though `C` has no way to load either of these.

This change updates the pre-compile ordering to be more parallel,
reflecting the true place where `Ext {A,B}` can be loaded:
```
  A       B
 / \     / \
C   Ext AB  |
 \    |    /
  \-- D --/
```

which allows `C` to compile in parallel with `B` and `Ext{A,B}`
@topolarity topolarity merged commit 6c5f221 into JuliaLang:master Nov 21, 2024
7 checks passed
topolarity added a commit that referenced this pull request Nov 21, 2024
…ies (#56624)

This is a slimmed-down version of #55910, without any of the
re-factoring. This improves the parallelism of the pre-compile job and
fixes a correctness bug, where unconditionally injecting a dependency
after its parent could be in conflict with dependencies in between
triggers, causing false extension cycles.

Without this fix, adding `AMDGPU.jl` and `Tracker.jl` reports a false
cycle on 1.11
@topolarity topolarity deleted the ct/fix-pkg-ext-deps branch November 21, 2024 04:15
@topolarity topolarity removed the backport 1.11 Change should be backported to release-1.11 label Nov 21, 2024
topolarity pushed a commit that referenced this pull request Nov 21, 2024
@KristofferC KristofferC mentioned this pull request Nov 22, 2024
30 tasks
@topolarity topolarity added backport 1.10 Change should be backported to the 1.10 release and removed backport 1.10 Change should be backported to the 1.10 release labels Nov 23, 2024
topolarity added a commit to topolarity/julia that referenced this pull request Nov 24, 2024
These are the main correctness fix from JuliaLang#55910, so it's important that
we have test coverage for it.
topolarity added a commit to topolarity/julia that referenced this pull request Nov 24, 2024
These are the main correctness fix from JuliaLang#55910, so it's important that
we have test coverage for it.
topolarity added a commit that referenced this pull request Nov 24, 2024
…56666)

It is possible for an extension `ExtAB` to be loadable by one of its
triggers, e.g. if `A` loads `B`. However, this loading is not supposed
to happen during pre-compilation of `A`.

Getting this wrong means disagreeing with the scheduled pre-compile jobs
(`A` is not scheduled to depend on or generate a cache file for `ExtAB`
but accidentally attempts both) and leads to confusing errors about
missing cache files.

We used to cover up this bad behavior w/ an erroneous cycle warning
(fixed by #55910), but now we need to be sure this works.
topolarity added a commit that referenced this pull request Nov 24, 2024
…56666)

It is possible for an extension `ExtAB` to be loadable by one of its
triggers, e.g. if `A` loads `B`. However, this loading is not supposed
to happen during pre-compilation of `A`.

Getting this wrong means disagreeing with the scheduled pre-compile jobs
(`A` is not scheduled to depend on or generate a cache file for `ExtAB`
but accidentally attempts both) and leads to confusing errors about
missing cache files.

We used to cover up this bad behavior w/ an erroneous cycle warning
(fixed by #55910), but now we need to be sure this works.
@topolarity topolarity removed the backport 1.10 Change should be backported to the 1.10 release label Nov 24, 2024
topolarity added a commit to topolarity/julia that referenced this pull request Nov 24, 2024
…uliaLang#56666)

It is possible for an extension `ExtAB` to be loadable by one of its
triggers, e.g. if `A` loads `B`. However, this loading is not supposed
to happen during pre-compilation of `A`.

Getting this wrong means disagreeing with the scheduled pre-compile jobs
(`A` is not scheduled to depend on or generate a cache file for `ExtAB`
but accidentally attempts both) and leads to confusing errors about
missing cache files.

We used to cover up this bad behavior w/ an erroneous cycle warning
(fixed by JuliaLang#55910), but now we need to be sure this works.
topolarity added a commit that referenced this pull request Nov 24, 2024
…n extensions (#56666) (#56676)

It is possible for an extension `ExtAB` to be loadable by one of its
triggers, e.g. if `A` loads `B`. However, this loading is not supposed
to happen during pre-compilation of `A`.

Getting this wrong means disagreeing with the scheduled pre-compile jobs
(`A` is not scheduled to depend on or generate a cache file for `ExtAB`
but accidentally attempts both) and leads to confusing errors about
missing cache files.

We used to cover up this bad behavior w/ an erroneous cycle warning
(fixed by #55910), but now we need to be sure this works.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix This change fixes an existing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants