Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backports for 1.11.2 #56228

Merged
merged 78 commits into from
Nov 21, 2024
Merged

Backports for 1.11.2 #56228

merged 78 commits into from
Nov 21, 2024

Conversation

KristofferC
Copy link
Member

@KristofferC KristofferC commented Oct 18, 2024

Backported PRs:

Need manual backport:

Contains multiple commits, manual intervention needed:

Non-merged PRs with backport label:

nsajko and others added 10 commits October 18, 2024 12:51
Other changes:
* replace `:total` with the less powerful `:foldable`
* add an `<:Integer` dispatch constraint on the `rationalize` method,
closes #55872
* replace `Rational{<:Integer}` with just `Rational`, they're equal

Other issues, related to `BigFloat` precision, are still present in
irrationals.jl, to be fixed by followup PRs, including #55853.

Fixes #55874

(cherry picked from commit d60837f)
Implementing `widen` isn't a requirement any more, since #26022.

(cherry picked from commit e95860c)
Avoids the

```
11: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for Base.Precompilation.ExplicitEnv(::String) (84 children)
```

shown in
#56080 (comment)

Co-authored-by: KristofferC <[email protected]>
(cherry picked from commit dc609a7)
Avoids the invalidation

```
   backedges: 1: superseding sizeof(s::AbstractString) @ Base strings/basic.jl:177 with MethodInstance for sizeof(::AbstractString) (75 children)
```

shown in
#56080 (comment).

Co-authored-by: KristofferC <[email protected]>
(cherry picked from commit 9844d85)
…b pkgimages (#54093)

Apparently (thanks ChatGPT) each line in a makefile is executed in a
separate shell so adding an `export` line on one line does not propagate
to the next line.

(cherry picked from commit b86e647)
Documenter.jl requires all trailing list content to follow the same
indentation as the header. So, in the current view
(https://docs.julialang.org/en/v1/manual/installation/#Command-line-arguments)
the list appears broken.

(cherry picked from commit 9f92989)
Closes #56109

Simulating a bad manifest by having `LibGit2_jll` missing as a dep of
`LibGit2` in my default env, say because the manifest was generated by a
different julia version or different master julia commit.

## This PR, it just works
```
julia> using Revise

julia>
```
i.e.
```
% JULIA_DEBUG=loading ./julia --startup-file=no
julia> using Revise
...
┌ Debug: Stdlib LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] is trying to load `LibGit2_jll`
│ which is not listed as a dep in the load path manifests, so resorting to search
│ in the stdlib Project.tomls for true deps
└ @ Base loading.jl:387
┌ Debug: LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] indeed depends on LibGit2_jll in project /Users/ian/Documents/GitHub/julia/usr/share/julia/stdlib/v1.12/LibGit2/Project.toml
└ @ Base loading.jl:395
...

julia>
```

## Master
```
julia> using Revise
Info Given Revise was explicitly requested, output will be shown live
ERROR: LoadError: ArgumentError: Package LibGit2 does not have LibGit2_jll in its dependencies:
- Note that the following manifests in the load path were resolved with a potentially
  different DEV version of the current version, which may be the cause of the error.
  Try to re-resolve them in the current version, or consider deleting them if that fails:
    /Users/ian/.julia/environments/v1.12/Manifest.toml
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have LibGit2 checked out for development and have
  added LibGit2_jll as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with LibGit2
...
```

(cherry picked from commit b02d671)
This removes several `convert(String, ...)` from this code, which really
shouldn't be something we invalidate on in the first place (see
#56173) but this is still an
improvement in code quality so let's take it.

(cherry picked from commit a7521ea)
@KristofferC KristofferC changed the title Backports for 1.11.3 Backports for 1.11.2 Oct 18, 2024
@giordano giordano added the release Release management and versioning. label Oct 18, 2024
topolarity and others added 11 commits October 18, 2024 13:55
…56159)

These are invalidated by our own stdlibs (Dates and REPL) unfortunately
so we need to put this barrier in.

This fix is _very_ un-satisfying, because it doesn't do anything to
solve this problem for downstream libraries that use e.g. `displaysize`.
To fix that, I think we need a way to make sure callers get these
invalidation barriers by default...

(cherry picked from commit 9223088)
…6089)

Fix https://github.com/JuliaLang/julia/issues/56085 by calling a newly
created `MulAddMul` object that only wraps the `alpha` (with `beta` set
to `false`). This avoids the explicit multiplication if `alpha` is known
to be `isone`.

(cherry picked from commit 0af99e6)
This bump contains the following commits:
a5b1174 * Adjust to change of annotations type in Base
d9d7472 * Adopt Base's annotated types/functions as API
da41b6a * Call load_customisations! automatically, lazily
af972e0 * Fix markdown syntax in "Mark API as experimental"
8332e45 * Mark API as Experimental
cfcfe8c * Fix syntax of generated CSS
c82409c * Remove tag wrapping from HTML show method
03211c9 * Small optimisation to single-interpolation styled
d080103 * Improve type stability within styled"" macro
448314b * More meticulously check indices in eachregion
02cd20b * Consistent Face hashes
This change by itself doesn't do anything significant on `master`, but
when backported to the v1.11 branch it'll address #56177. However it'd
be great if someone could tell _why_ this fixes that issue, because it
looks very unrelated.

---------

Co-authored-by: Ian Butterworth <[email protected]>
(cherry picked from commit f36f342)
Adds a new internal function `_take!(dst::Array{T,N}, src::Array{T,N})` for
doing an efficient `copyto!` equivalent. Previously it was assumed that
`compact` did this automatically, which wasn't a great assumption.

Fixes #56078

(cherry picked from commit fc40e62)
Falling back to the older serial precompilation process is basically a
bug (except for if a manifest hasn't been resolved) so
#52619 added more info on why
it's been hit so we have a chance of fixing issues that are otherwise
very difficult to recreate.

However "invalid header" which usually just means it was made by a
different julia version appears to sound too alarming to users.
https://discourse.julialang.org/t/cache-misses-when-using-packages-since-upgrading-to-1-11/121445

So soften it there and in error messages, given it seems a better
description.

Suggested by @giordano in
https://discourse.julialang.org/t/cache-misses-when-using-packages-since-upgrading-to-1-11/121445/4?u=ianshmean

(cherry picked from commit aa51abe)
…st_throws (#56231)

Fix #54082

Arguably this was a breaking change (as a consequence of
#51979). But regardless, it seems
like useful functionality to have a public API for testing that an
`UndefVarError` was thrown for the expected variable name (regardless of
scope). This is particularly useful if you don't know what the scope is
(for example, in my use-case i want to test that a specific
`UndefVarError` is thrown from a module with a `gensym`'d name).

Pre-v1.11 the syntax for this was
```julia
@test_throws UndefVarError(:x) foo()
```
but that stopped working in v1.11 when `UndefVarError` got a second
field (in fact in v1.11.1 this is an error when before it would pass)

This PR restores that functionality.

We might want to backport it to v1.11.x so that v1.11 isn't the only
version that doesn't support this.

(cherry picked from commit b0c1525)
We currently never remove asserts but I still think this is not a good
practice.

(cherry picked from commit 8e67f99)
Signed-off-by: alongdate <[email protected]>
(cherry picked from commit 2f90962)
@IanButterworth
Copy link
Member

IanButterworth commented Oct 20, 2024

I tried to cherry-pick #56214 here but it's a bit tricky because precompilepkgs has changed a fair bit with the workspaces support, which isn't on 1.11. (I did add two minor backports that help reduce the merge conflicts though.) I highlight that as I think it's something we'd want to get in here. cc. @vtjnash

IanButterworth and others added 3 commits October 20, 2024 13:35
…b6e99 (#56261)

Stdlib: Pkg
URL: https://github.com/JuliaLang/Pkg.jl.git
Stdlib branch: release-1.11
Julia branch: backports-release-1.11
Old commit: aba90d22b
New commit: 9438b6e99
Julia version: 1.11.1
Pkg version: 1.11.0(Does not match)
Bump invoked by: @IanButterworth
Powered by:
[BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl)

Diff:
JuliaLang/Pkg.jl@aba90d2...9438b6e

```
$ git log --oneline aba90d22b..9438b6e99
9438b6e99 Merge pull request #4055 from JuliaLang/backports-release-1.11
c83160aac REPLExt: use Base.isaccessibledir rather than isdir in completions (#4053)
7997ec320 REPLExt: run repl hint generation for modeswitch chars when not switching (#4054)
```

Co-authored-by: Dilum Aluthge <[email protected]>
@KristofferC
Copy link
Member Author

@IanButterworth, I attempted a cherry-pick as well. It seems to work ok.

@KristofferC
Copy link
Member Author

@maleadt, seems like the llvmpasses test fails after backporting 9c94a38

KristofferC and others added 7 commits November 11, 2024 12:48
#56369)

Before (in an environment with DifferentialEquations.jl):

```julia
julia> @time Pkg.precompile()
  0.733576 seconds (3.44 M allocations: 283.676 MiB, 6.24% gc time)

julia> isfile_calls[1:10]
10-element Vector{Pair{String, Int64}}:
        "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Printf/3FQLY_zHycD.ji" => 178
        "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Printf/3FQLY_xxrt3.ji" => 178
         "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Dates/p8See_xxrt3.ji" => 158
         "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Dates/p8See_zHycD.ji" => 158
          "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/TOML/mjrwE_zHycD.ji" => 155
          "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/TOML/mjrwE_xxrt3.ji" => 155
                                     "/home/kc/.julia/compiled/v1.12/Preferences/pWSk8_4Qv86.ji" => 152
                                     "/home/kc/.julia/compiled/v1.12/Preferences/pWSk8_juhqb.ji" => 152
 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_zHycD.ji" => 144
 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_xxrt3.ji" => 144
 ```

After:

```julia
julia> @time Pkg.precompile()
  0.460077 seconds (877.59 k allocations: 108.075 MiB, 4.77% gc time)

julia> isfile_calls[1:10]
  10-element Vector{Pair{String, Int64}}:
"/tmp/jl_a5xFWK/Project.toml" => 15
"/tmp/jl_a5xFWK/Manifest.toml" => 7
"/home/kc/.julia/registries/General.toml" => 6

"/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Markdown/src/Markdown.jl"
=> 3

"/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl"
=> 3

"/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/Distributed.jl"
=> 3

"/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/UUIDs/src/UUIDs.jl"
=> 3

"/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/LibCURL/src/LibCURL.jl"
=> 3
```

Performance is improved and we are not calling `isfile` on a bunch of the same ji files hundreds times.

Benchmark is made on a linux machine so performance diff should be a lot better on Windows where these `isfile_casesensitive` call is much more expensive.

Fixes #56366

---------

Co-authored-by: KristofferC <[email protected]>
Co-authored-by: Ian Butterworth <[email protected]>
(cherry picked from commit 9850a38)
Follow up to #56392: also the warning in
https://docs.julialang.org/en/v1.11.1/devdocs/llvm-passes/#Multiversioning
is rendered incorrectly because of a missing space.

(cherry picked from commit 3de1b1d)
currentlu this route is mentioned in docs
https://docs.julialang.org/en/v1/stdlib/Profile/#Triggered-During-Execution
but missing from the module docstring, this should help users who have
little idea how to "send a kernel signal to a process" to get started

---------

Co-authored-by: Ian Butterworth <[email protected]>
(cherry picked from commit 9af0dea)
Fixes #56141
Introduced by #52228 (a624d44)

(cherry picked from commit 671cd5e)
Co-authored-by: Viral B. Shah <[email protected]>
(cherry picked from commit cd748a5)
@KristofferC
Copy link
Member Author

Okay, let's run another PkgEval and then not merge too much in here so we can fix the possible regressions and get this out.

@nanosoldier runtests()

@nanosoldier
Copy link
Collaborator

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

If we expect this to be a public API
(https://github.com/timholy/Revise.jl for some reason is trying to
access this state), we should lock around it for consistency with the
other similar functions.

Needed for timholy/Revise.jl#856

(cherry picked from commit 4cdd864)
@KristofferC
Copy link
Member Author

I think the latest backport should fix Revise

@KristofferC
Copy link
Member Author

@nanosoldier runtests(["JuliaSyntaxHighlighting", "JuliaInterpreter", "LogDensityProblemsAD", "Revise", "FromFile", "Handcalcs", "CharacteristicInvFourier", "BenchmarkProfiles", "AxisKeysExtra", "RegressionTables", "ExtendableGrids", "MCVI", "StipplePlotly", "StipplePlotlyExport", "NonconvexMultistart", "PlutoPages", "ParticleInCell", "DistributedFactorGraphs", "WaveSpec", "FrequencySweep", "SpiDy"])

@IanButterworth
Copy link
Member

Is #56077 fixed on this with the latest backports?

@KristofferC
Copy link
Member Author

Doesn't look like it, the PR that fixes it seems quite big to backport...

@tecosaur
Copy link
Contributor

Kristoffer in another PR (but intended for here):

looks like JuliaSyntaxHighligting needs updating to work with this.

JuliaSyntaxHighligting shouldn't be included as a 1.11 stdlib though? Unless you're talking about the general-released package, which indeed needs me to backport the changes to a new release.

@KristofferC
Copy link
Member Author

Unless you're talking about the general-released package, which indeed needs me to backport the changes to a new release.

Yes.

@dkarrasch
Copy link
Member

Before releasing v1.11.2, i.e., after we merge this here, we should first merge JuliaSparse/SparseArrays.jl#571 and then bump SparseArrays.jl on the release-v1.11 branch.

@nanosoldier
Copy link
Collaborator

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

KristofferC and others added 4 commits November 14, 2024 09:00
Manual "backport" of #56538. I assume we want this against
#56228 ?
…to 2420351 (#56608)

Stdlib: SparseArrays
URL: https://github.com/JuliaSparse/SparseArrays.jl.git
Stdlib branch: release-1.11
Julia branch: backports-release-1.11
Old commit: cb602d7
New commit: 2420351
Julia version: 1.11.1
SparseArrays version: 1.11.0(Does not match)
Bump invoked by: @dkarrasch
Powered by:
[BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl)

Diff:
JuliaSparse/SparseArrays.jl@cb602d7...2420351

```
$ git log --oneline cb602d7..2420351
2420351 Allow for duplicate docstrings in linalg docs (#584)
0a1c886 Merge pull request #571 from JuliaSparse/backports-release-1.11
298f5e1 Break recursion (#579)
7bc65ab Update CI
fcc6102 Do not use nested dissection by default. (#550)
8534357 Update ci.yml: run CI and docs on v1.11
66d65fd Change default QR tolerance to match SPQR (#557)
8a48327 test: Don't use GPL module when Base.USE_GPL_LIBS=false (#535)
8dd8300 SparseMatrixCSC constructor with a Tuple of Integers (#523)
aa66624 doc: move solvers doc to `src\solvers.md` (#576)
8a85fbe Inline sparse-times-dense in-place multiplication
```

Co-authored-by: Dilum Aluthge <[email protected]>
…ies (#56624)

This is a slimmed-down version of #55910, without any of the
re-factoring. This improves the parallelism of the pre-compile job and
fixes a correctness bug, where unconditionally injecting a dependency
after its parent could be in conflict with dependencies in between
triggers, causing false extension cycles.

Without this fix, adding `AMDGPU.jl` and `Tracker.jl` reports a false
cycle on 1.11
@KristofferC KristofferC merged commit 229f027 into release-1.11 Nov 21, 2024
6 checks passed
@KristofferC KristofferC deleted the backports-release-1.11 branch November 21, 2024 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release Release management and versioning.
Projects
None yet
Development

Successfully merging this pull request may close these issues.