Make world-age increments explicit #56509

Keno · 2024-11-09T04:40:10Z

This PR introduces a new, toplevel-only, syntax form :worldinc that semantically represents the effect of raising the current task's world age to the latest world for the remainder of the current toplevel evaluation (that context being an entry to eval or a module expression). For detailed motivation on why this is desirable, see #55145, which I won't repeat here, but the gist is that we never really defined when world-age increments and worse are inconsistent about it. This is something we need to figure out now, because the bindings partition work will make world age even more observable via bindings.

Having created a mechanism for world age increments, the big question is one of policy, i.e. when should these world age increments be inserted.

Several reasonable options exist:

After world-age affecting syntax constructs (as proprosed in Make precise semantics of world-age increments in top-level thunks #55145)
Option 1 + some reasonable additional cases that people rely on
Before any top level call expression
Before any expression at toplevel whatsover

As an example, case, consider a == a at toplevel. Depending on the semantics that could either be the same as in local scope, or each of the four world age dependent lookups (three binding lookups, one method lookup) could (potentially) occur in a different world age.

The general tradeoff here is between the risk of exposing the user to confusing world age errors and our ability to optimize top-level code (in general, any :worldinc statement will require us to fully pessimize or recompile all following code).

This PR basically implements option 2 with the following semantics:

The interpreter explicit raises the world age only at :worldinc exprs or after :module exprs.
The frontend inserts :worldinc after all struct definitions, method definitions, using and `import.
The @eval macro inserts a worldinc following the call to eval if at toplevel
A literal (syntactic) call to include gains an implicit worldinc.

Of these the fourth is probably the most questionable, but is necessary to make this non-breaking for most code patterns. Perhaps it would have been better to make include a macro from the beginning (esp because it already has semantics that look a little like reaching into the calling module), but that ship has sailed.

Unfortunately, I don't see any good intermediate options between this PR and option #3 above. I think option #3 is closest to what we have right now, but if we were to choose it and actually fix the soundness issues, I expect that we would be destroying all performance of global-scope code. For this reason, I would like to try to make the version in this PR work, even if the semantics are a little ugly.

The biggest pattern that this PR does not catch is:

eval(:(f() = 1))
f()

We could apply the same include special case to eval, but given the existence of @eval which allows addressing this at the macro level, I decided not to. We can decide which way we want to go on this based on what the package ecosystem looks like.

Keno · 2024-11-09T08:25:39Z

The SparseArrays test failure is known - SparseArrays.allowscalar redefines indexing methods and the test relies on that getting picked up right away. Should be changed to a macro with a worldage increment probably, but before going through the hassle of the multi repo coordination, let's finish this PR.

Keno · 2024-11-09T08:25:48Z

@nanosoldier runtests()

aviatesk · 2024-11-09T11:54:26Z

I’m not opposed to this PR, but what’s the specific reason for introducing :worldinc? From what I understand from reading #55145, it seems like modifying the runtime to adhere to these rules would avoid the discussed issues?

We stop incrementing the world age for every statement. Instead, world age (at the toplevel) is incremented to the latest world age only at the following points:
a. Entry into toplevel evaluation (i.e., at the beginning of Core.eval)
b. After any evaluation of :method
c. On the bindings branch, after introducing a new binding

(plus an increment after a :module expression as per you mentioned above)

That said, I think introducing :worldinc would make it easier to fix inference-related problems, like ensuring world-age is incremented correctly for include as done in this PR. So I’m actually in favor of adding this expression type, but I’d like to understand the reasoning clearly.

aviatesk · 2024-11-09T11:55:56Z

test/reflection.jl

@@ -777,6 +777,7 @@ faz1(x) = foo1(x)
 @test faz1(1.0) == 1
 m = first(methods(bar1, Tuple{Int}))
 Base.delete_method(m)


Does this mean that the result of Base.delete_method (and Base.delete_binding) won’t take effect until the user explicitly calls Core.@worldinc? That sounds like it would be a pretty breaking change. I agree with the idea of restricting the effect of :worldinc to the top level (for helping inference), but it seems like Base.delete_method might need some adjustments, such as calling @eval internally or something similar.

Does this mean that the result of Base.delete_method (and Base.delete_binding) won’t take effect until the user explicitly calls Core.@WorldInc?

It takes effect immediately (in a new world age), but the effect is not visible in the current task until it raises it world age.

Calling @eval internally will not raise the world age, because it's not at top level. delete_method is mostly used internally by Revise which can take an appropriate adjustment. There could also be a delete_method macro that does the world age raising implicitly.

Keno · 2024-11-10T02:20:38Z

but what’s the specific reason for introducing :worldinc?

Two reasons:

As a separation of concerns issue and to put the more complicated world-age increment decisions into lowering rather than forcing the logic to be duplicated in several places
To enable it to be inserted by macros that may want it

Keno · 2024-11-10T02:25:05Z

@vtjnash suggests implicit world-age increments between the args of :toplevel and :module. I think this is reasonable and I'll make an appropriate adjustment.

Keno · 2024-11-10T02:58:03Z

@vtjnash suggests implicit world-age increments between the args of :toplevel and :module.

Forgot to mention that @vtjnash pointed out that I already made world age increment for macro purposes in #53515, so it would be inconsistent for it not to increment the world age for execution, which I found convincing.

nanosoldier · 2024-11-10T03:37:42Z

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Keno · 2024-11-10T03:44:29Z

@nanosoldier runtests(["Tricks", "MethodInspector", "ExprTools", "OptimizingIR", "OverflowContexts", "CBOOCall", "LazyModules", "PowerSeries", "Purses", "Hygienic", "MullerPlot", "SyntaxTree", "Plugins", "ParameterisedModule", "ApproximationAnalysis", "VectorInterface", "Expronicon", "TestExtras", "Hyperspecialize", "TimerOutputs", "Behavior", "ProblemSet", "FinitePosets", "CommandLiner", "TrixiBase", "IRTools", "TypeStability", "LoweredCodeUtils", "LispSyntax", "FileIO", "Automa", "ComoniconTargetExpr", "SplittablesBase", "HAML", "JSON3", "PDDL", "Continuables", "DataSets", "DataFlowTasks", "GraphQLGen", "Revise", "ConfigurationsENV", "DebugAdapter", "LIBSVM", "FromFile", "Groups", "LazyReports", "Polyester", "Handcalcs", "PkgJogger", "IntervalLinearAlgebra", "TypeClasses", "Open62541", "MHLib", "ExtensibleEffects", "Arblib", "PlutoVista", "LegendrePolynomials", "LinkedInAPI", "SwagUI", "GraphQLClient", "Folds", "Serde", "FlatRBAC", "BenchmarkProfiles", "DataToolkitCommon", "ACSets", "SphericalHarmonicExpansions", "CrystalInfoFramework", "Experimenter", "MATDaemon", "DrelTools", "DistributedSparseGrids", "TransitionalMCMC", "AstroRepresentations", "TransformSpecifications", "PlutoPlotly", "Bukdu", "OndaEDFSchemas", "AppleHealthParser", "MultivariateChebyshev", "Onda", "AlignedSpans", "YasolSolver", "JSXGraph", "Semagrams", "Gen", "MaterialDecomposition", "CSetAutomorphisms", "GeniePlugins", "GenieSession", "GeniePackageManager", "GenieCache", "GenieDeployHeroku", "GenieCacheFileCache", "GenieDeployDocker", "GenieSessionFileSession", "InfiniteOpt", "EnergyModelsBase", "EnergyModelsCO2", "EnergyModelsRenewableProducers", "EnergyModelsGeography", "OndaEDF", "MembraneBase", "Books", "StippleMathjs", "Stipple", "StippleUI", "SparseArrayKit", "QWignerSymbols", "RegNets", "GenieDevTools", "HerbSearch", "Equate", "StippleCharts", "Herb", "StipplePlotly", "SteadyStateDiffEq", "StipplePlotlyExport", "TaylorInversion", "GenieAuthorisation", "CategoryData", "GenieFramework", "ReversePropagation", "GivEmXL", "GenericCharacterTables", "SphericalFunctions", "StippleKeplerGL", "Spehulak", "PlutoPages", "GeometryOptimization", "ProToPortal", "JuliaBUGS", "MLJTestIntegration", "CausalGPSLC", "MLJ", "WaveSpec", "Individual", "AntennaPattern", "SimpleCrop", "UnfoldBIDS", "LeafGasExchange", "Garlic", "EMpht", "Vahana", "Petri", "Lighthouse", "AffineMotions", "PlantGeomTurtle", "HetaSimulator", "Serialization", "CropRootBox"])

nanosoldier · 2024-11-10T05:33:05Z

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Keno · 2024-11-11T00:20:57Z

Alright, a fair number of tests with dependencies on JuliaInterpreter or Revise, which need to be taught about :worldinc, but also some that noticed the behavior change. I think the thing to do is

an independent PR that just introduces :worldinc, but doesn't use it yet
Fix JuliaInterpreter/Revise
Re-run PkgEval and see what to do about the rest of the packages

Split out from #56509 to facilitate adjusting downstream packages.

This was part of #56509, but is an independent bugfix. The basic issue is that these macro were using `do` block internally. This is undesirable for test macros, because we would like them not to affect the behavior of what they're testing. E.g. right now: ``` julia> using Test julia> const x = 1 1 julia> @test_nowarn const x = 1 ERROR: syntax: `global const` declaration not allowed inside function around /home/keno/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:927 Stacktrace: [1] top-level scope @ REPL[3]:1 ``` This PR just writes out the try/finally manually, so the above works fine after this PR.

Split out from #56509 to facilitate adjusting downstream packages.

vtjnash · 2024-11-13T19:16:51Z

stdlib/REPL/src/REPL.jl

@@ -367,6 +367,7 @@ function eval_user_input(@nospecialize(ast), backend::REPLBackend, mod::Module)
                for xf in backend.ast_transforms
                    ast = Base.invokelatest(xf, ast)
                end
+                toplevel_eval_with_hooks(mod, Expr(:worldinc))


This is a no-op (if the implementation is not broken)

Suggested change

toplevel_eval_with_hooks(mod, Expr(:worldinc))

vtjnash

Overall SGTM (though needs a rebase against the parts that are merged already).

Looks like a lot of the needed added Core.@worldinc also will be already solved by making each toplevel / module statement continue to imply an updated world (since they already imply an updated macro world, and the statements are supposed to run in the world after that macro expansion, which wouldn't be possible to express with these explicit markers anyways)

One other remaining todo item is adding this to the Compiler verifier code. I don't know if we run any debug tests on PRs, but I believe we run those on nightly, and it likely should error on the unexpected Expr head.

These were added in JuliaLang/julia#56523. This simply adds tracking for the world age to the interpreter, but does not use it for anything. JuliaInterpreter's current model of world age does not match Base anyway. We should fix that eventually, but it's probably easier to do that once JuliaLang/julia#56509 and binding partitions are merged.

* Handle `:latestworld` expr These were added in JuliaLang/julia#56523. This simply adds tracking for the world age to the interpreter, but does not use it for anything. JuliaInterpreter's current model of world age does not match Base anyway. We should fix that eventually, but it's probably easier to do that once JuliaLang/julia#56509 and binding partitions are merged. * 1.6 compat

Upcoming base PR JuliaLang/julia#56509 makes precise when exactly world age increments automatically at top-level. Packages are mostly not supposed to notice, but of course Revise is a bit patholgical in that it does basically expect asynchronous changes to have top level effects. After that PR, that requires an explicit opt-in. This PR makes the appropriate adjustments to the tests.

Keno · 2024-11-18T03:33:23Z

@nanosoldier runtests(["Tricks", "MethodInspector", "ExprTools", "OptimizingIR", "OverflowContexts", "CBOOCall", "LazyModules", "PowerSeries", "Purses", "Hygienic", "MullerPlot", "SyntaxTree", "Plugins", "ParameterisedModule", "ApproximationAnalysis", "VectorInterface", "Expronicon", "TestExtras", "Hyperspecialize", "TimerOutputs", "Behavior", "ProblemSet", "FinitePosets", "CommandLiner", "TrixiBase", "IRTools", "TypeStability", "LoweredCodeUtils", "LispSyntax", "FileIO", "Automa", "ComoniconTargetExpr", "SplittablesBase", "HAML", "JSON3", "PDDL", "Continuables", "DataSets", "DataFlowTasks", "GraphQLGen", "Revise", "ConfigurationsENV", "DebugAdapter", "LIBSVM", "FromFile", "Groups", "LazyReports", "Polyester", "Handcalcs", "PkgJogger", "IntervalLinearAlgebra", "TypeClasses", "Open62541", "MHLib", "ExtensibleEffects", "Arblib", "PlutoVista", "LegendrePolynomials", "LinkedInAPI", "SwagUI", "GraphQLClient", "Folds", "Serde", "FlatRBAC", "BenchmarkProfiles", "DataToolkitCommon", "ACSets", "SphericalHarmonicExpansions", "CrystalInfoFramework", "Experimenter", "MATDaemon", "DrelTools", "DistributedSparseGrids", "TransitionalMCMC", "AstroRepresentations", "TransformSpecifications", "PlutoPlotly", "Bukdu", "OndaEDFSchemas", "AppleHealthParser", "MultivariateChebyshev", "Onda", "AlignedSpans", "YasolSolver", "JSXGraph", "Semagrams", "Gen", "MaterialDecomposition", "CSetAutomorphisms", "GeniePlugins", "GenieSession", "GeniePackageManager", "GenieCache", "GenieDeployHeroku", "GenieCacheFileCache", "GenieDeployDocker", "GenieSessionFileSession", "InfiniteOpt", "EnergyModelsBase", "EnergyModelsCO2", "EnergyModelsRenewableProducers", "EnergyModelsGeography", "OndaEDF", "MembraneBase", "Books", "StippleMathjs", "Stipple", "StippleUI", "SparseArrayKit", "QWignerSymbols", "RegNets", "GenieDevTools", "HerbSearch", "Equate", "StippleCharts", "Herb", "StipplePlotly", "SteadyStateDiffEq", "StipplePlotlyExport", "TaylorInversion", "GenieAuthorisation", "CategoryData", "GenieFramework", "ReversePropagation", "GivEmXL", "GenericCharacterTables", "SphericalFunctions", "StippleKeplerGL", "Spehulak", "PlutoPages", "GeometryOptimization", "ProToPortal", "JuliaBUGS", "MLJTestIntegration", "CausalGPSLC", "MLJ", "WaveSpec", "Individual", "AntennaPattern", "SimpleCrop", "UnfoldBIDS", "LeafGasExchange", "Garlic", "EMpht", "Vahana", "Petri", "Lighthouse", "AffineMotions", "PlantGeomTurtle", "HetaSimulator", "Serialization", "CropRootBox"])

Goes along with JuliaLang/julia#56509

base/essentials.jl

base/tuple.jl

src/codegen.cpp

src/toplevel.c

src/interpreter.c

src/codegen.cpp

Compiler/src/inferencestate.jl

test/ccall.jl

test/core.jl

test/error.jl

This test currently relies on implicit world age increments at top level. We're re-evaluating where these go because julia is currently inconsistent about it in the interpreter, compiler and inference. To make sure this test keeps working on 1.12, add an explicit world age increment. See JuliaLang/julia#56509.

- simplify float.jl loop not to call functions just defined to get back the value just stored there - add method to the correct checkbounds function (instead of a local) - missing world push/pop around jl_interpret_toplevel_thunk after #56509 - jl_lineno use maybe missed in #53799 - add some debugging dump for scm_to_julia_ mistakes

* Implement faster `issubset` for `CartesianIndices{N}` (#56282) Co-authored-by: xili <[email protected]> * Improve doc example: Extracting the type parameter from a super-type (#55983) Documentation describes the correct way of extracting the element type of a supertype: https://docs.julialang.org/en/v1/manual/methods/#Extracting-the-type-parameter-from-a-super-type However, one of the examples to showcase this is nonsensical since it is a union of multiple element types. I have replaced this example with a union over the dimension. Now, the `eltype_wrong` function still gives a similar error, yet the correct way returns the unambiguous answer. --------- Co-authored-by: Lilith Orion Hafner <[email protected]> * llvmpasses: force vector width for compatibility with non-x86 hosts. (#56300) The pipeline-prints test currently fails when running on an aarch64-macos device: ``` /Users/tim/Julia/src/julia/test/llvmpasses/pipeline-prints.ll:309:23: error: AFTERVECTORIZATION: expected string not found in input ; AFTERVECTORIZATION: vector.body ^ <stdin>:2:40: note: scanning from here ; *** IR Dump Before AfterVectorizationMarkerPass on julia_f_199 *** ^ <stdin>:47:27: note: possible intended match here ; *** IR Dump Before AfterVectorizationMarkerPass on jfptr_f_200 *** ^ Input file: <stdin> Check file: /Users/tim/Julia/src/julia/test/llvmpasses/pipeline-prints.ll -dump-input=help explains the following input dump. Input was: <<<<<< 1: opt: WARNING: failed to create target machine for 'x86_64-unknown-linux-gnu': unable to get target for 'x86_64-unknown-linux-gnu', see --version and --triple. 2: ; *** IR Dump Before AfterVectorizationMarkerPass on julia_f_199 *** check:309'0 X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found 3: define i64 @julia_f_199(ptr addrspace(10) noundef nonnull align 16 dereferenceable(40) %0) #0 !dbg !4 { check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4: top: check:309'0 ~~~~~ 5: %1 = call ptr @julia.get_pgcstack() check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 6: %ptls_field = getelementptr inbounds ptr, ptr %1, i64 2 check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7: %ptls_load45 = load ptr, ptr %ptls_field, align 8, !tbaa !8 check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ . . . 42: check:309'0 ~ 43: L41: ; preds = %L41.loopexit, %L17, %top check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 44: %value_phi10 = phi i64 [ 0, %top ], [ %7, %L17 ], [ %.lcssa, %L41.loopexit ] check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 45: ret i64 %value_phi10, !dbg !52 check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 46: } check:309'0 ~~ 47: ; *** IR Dump Before AfterVectorizationMarkerPass on jfptr_f_200 *** check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ check:309'1 ? possible intended match 48: ; Function Attrs: noinline optnone check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 49: define nonnull ptr addrspace(10) @jfptr_f_200(ptr addrspace(10) %0, ptr noalias nocapture noundef readonly %1, i32 %2) #1 { check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 50: top: check:309'0 ~~~~~ 51: %3 = call ptr @julia.get_pgcstack() check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 52: %4 = getelementptr inbounds ptr addrspace(10), ptr %1, i32 0 check:309'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ . . . >>>>>> -- ******************** Failed Tests (1): Julia :: pipeline-prints.ll ``` The problem is that these tests assume x86_64, which fails because the target isn't available, so it presumably uses the native target which has different vectorization characteristics: ``` ❯ ./usr/tools/opt --load-pass-plugin=libjulia-codegen.dylib -passes='julia' --print-before=AfterVectorization -o /dev/null ../../test/llvmpasses/pipeline-prints.ll ./usr/tools/opt: WARNING: failed to create target machine for 'x86_64-unknown-linux-gnu': unable to get target for 'x86_64-unknown-linux-gnu', see --version and --triple. ``` There's other tests that assume this (e.g. the `fma` cpufeatures one), but they don't fail, so I've left them as is. * Reduce generic matrix*vector latency (#56289) ```julia julia> using LinearAlgebra julia> A = rand(Int,4,4); x = rand(Int,4); y = similar(x); julia> @time mul!(y, A, x, 2, 2); 0.330489 seconds (792.22 k allocations: 41.519 MiB, 8.75% gc time, 99.99% compilation time) # master 0.134212 seconds (339.89 k allocations: 17.103 MiB, 15.23% gc time, 99.98% compilation time) # This PR ``` Main changes: - `generic_matvecmul!` and `_generic_matvecmul!` now accept `alpha` and `beta` arguments instead of `MulAddMul(alpha, beta)`. The methods that accept a `MulAddMul(alpha, beta)` are also retained for backward compatibility, but these now forward `alpha` and `beta`, instead of the other way around. - Narrow the scope of the `@stable_muladdmul` applications. We now construct the `MulAddMul(alpha, beta)` object only where it is needed in a function call, and we annotate the call site with `@stable_muladdmul`. This leads to smaller branches. - Create a new internal function with methods for the `'N'`, `'T'` and `'C'` cases, so that firstly, there's less code duplication, and secondly, the `_generic_matvecmul!` method is now simple enough to enable constant propagation. This eliminates the unnecessary branches, and only the one that is taken is compiled. Together, this reduces the TTFX substantially. * Type `Base.is_interactive` as `Bool` (#56303) Before, typing `Base.is_interactive = 7` would cause weird internal REPL failures down the line. Now, it throws an InexactError and has no impact. * REPL: don't complete str and cmd macros when the input matches the internal name like `r_` to `r"` (#56254) * fix REPL test if a "juliadev" directory exists in home (#56218) * Fix trampoline warning on x86 as well (#56280) * typeintersect: more fastpath to skip intersect under circular env (#56304) fix #56040 * Preserve type in `first` for `OneTo` (#56263) With this PR, ```julia julia> first(Base.OneTo(10), 4) Base.OneTo(4) ``` Previously, this would have used indexing to return a `UnitRange`. This is probably the only way to slice a `Base.OneTo` and obtain a `Base.OneTo` back. * Matmul: dispatch on specific blas paths using an enum (#55002) This expands on the approach taken by https://github.com/JuliaLang/julia/pull/54552. We pass on more type information to `generic_matmatmul_wrapper!`, which lets us convert the branches to method dispatches. This helps spread the latency around, so that instead of compiling all the branches in the first call, we now compile the branches only when they are actually taken. While this reduces the latency in individual branches, there is no reduction in latency if all the branches are reachable. ```julia julia> A = rand(2,2); julia> @time A * A; 0.479805 seconds (809.66 k allocations: 40.764 MiB, 99.93% compilation time) # 1.12.0-DEV.806 0.346739 seconds (633.17 k allocations: 31.320 MiB, 99.90% compilation time) # This PR julia> @time A * A'; 0.030413 seconds (101.98 k allocations: 5.359 MiB, 98.54% compilation time) # v1.12.0-DEV.806 0.148118 seconds (219.51 k allocations: 11.652 MiB, 99.72% compilation time) # This PR ``` The latency is spread between the two calls here. In fresh sessions: ```julia julia> A = rand(2,2); julia> @time A * A'; 0.473630 seconds (825.65 k allocations: 41.554 MiB, 99.91% compilation time) # v1.12.0-DEV.806 0.490305 seconds (774.87 k allocations: 38.824 MiB, 99.90% compilation time) # This PR ``` In this case, both the `syrk` and `gemm` branches are reachable, so there is no reduction in latency. Analogously, there is a reduction in latency in the second set of matrix multiplications where we call `symm!/hemm!` or `_generic_matmatmul`: ```julia julia> using LinearAlgebra julia> A = rand(2,2); julia> @time Symmetric(A) * A; 0.711178 seconds (2.06 M allocations: 103.878 MiB, 2.20% gc time, 99.98% compilation time) # v1.12.0-DEV.806 0.540669 seconds (904.12 k allocations: 43.576 MiB, 2.60% gc time, 97.36% compilation time) # This PR ``` * Scaling `mul!` for generic `AbstractArray`s (#56313) This improves performance in the scaling `mul!` for `StridedArray`s by using loops instead of broadcasting. ```julia julia> using LinearAlgebra julia> A = zeros(200,200); C = similar(A); julia> @btime mul!($C, $A, 1, 2, 2); 19.180 μs (0 allocations: 0 bytes) # nightly v"1.12.0-DEV.1479" 11.361 μs (0 allocations: 0 bytes) # This PR ``` The latency is reduced as well for the same reason. ```julia julia> using LinearAlgebra julia> A = zeros(2,2); C = similar(A); julia> @time mul!(C, A, 1, 2, 2); 0.203034 seconds (522.94 k allocations: 27.011 MiB, 14.95% gc time, 99.97% compilation time) # nightly 0.034713 seconds (59.16 k allocations: 2.962 MiB, 99.91% compilation time) # This PR ``` Thirdly, I've replaced the `.*ₛ` calls by explicit branches. This fixes the following: ```julia julia> A = [zeros(2), zeros(2)]; C = similar(A); julia> mul!(C, A, 1) ERROR: MethodError: no method matching +(::Vector{Float64}, ::Bool) ``` After this, ```julia julia> mul!(C, A, 1) 2-element Vector{Vector{Float64}}: [0.0, 0.0] [0.0, 0.0] ``` Also, I've added `@stable_muladdmul` annotations to the `generic_mul!` call, but moved it within the loop to narrow its scope. This doesn't increase the latency, while making the call type-stable. ```julia julia> D = Diagonal(1:2); C = similar(D); julia> @time mul!(C, D, 1, 2, 2); 0.248385 seconds (898.18 k allocations: 47.027 MiB, 12.30% gc time, 99.96% compilation time) # nightly 0.249940 seconds (919.80 k allocations: 49.128 MiB, 11.36% gc time, 99.99% compilation time) # This PR ``` * InteractiveUtils.jl: fixes issue where subtypes resolves bindings and causes deprecation warnings (#56306) The current version of `subtypes` will throw deprecation errors even if no one is using the deprecated bindings. A similar bug was fixed in Aqua.jl - https://github.com/JuliaTesting/Aqua.jl/pull/89/files See discussion here: - https://github.com/JuliaIO/ImageMagick.jl/issues/235 (for identifying the problem) - https://github.com/simonster/Reexport.jl/issues/42 (for pointing to the issue in Aqua.jl) - https://github.com/JuliaTesting/Aqua.jl/pull/89/files (for the fix in Aqua.jl) This adds the `isbindingresolved` test to the `subtypes` function to avoid throwing deprecation warnings. It also adds a test to check that this doesn't happen. --- On the current master branch (before the fix), the added test shows: ``` WARNING: using deprecated binding InternalModule.MyOldType in OuterModule. , use MyType instead. Subtypes and deprecations: Test Failed at /home/dgleich/devextern/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:932 Expression: isempty(stderr_content) Evaluated: isempty("WARNING: using deprecated binding InternalModule.MyOldType in OuterModule.\n, use MyType instead.\n") Test Summary: | Fail Total Time Subtypes and deprecations | 1 1 2.8s ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken. in expression starting at /home/dgleich/devextern/julia/stdlib/InteractiveUtils/test/runtests.jl:841 ERROR: Package InteractiveUtils errored during testing ``` --- Using the results of this pull request: ``` @test_nowarn subtypes(Integer); ``` passes without error. The other tests pass too. * [CRC32c] Support AbstractVector{UInt8} as input (#56164) This is a similar PR to https://github.com/JuliaIO/CRC32.jl/pull/12 I added a generic fallback method for `AbstractVector{UInt8}` similar to the existing generic `IO` method. Co-authored-by: Steven G. Johnson <[email protected]> * Put `jl_gc_new_weakref` in a header file again (#56319) * use textwidth for string display truncation (#55442) It makes a big difference when displaying strings that have width-2 or width-0 characters. * Use `pwd()` as the default directory to walk in `walkdir` (#55550) * Reset mtime of BOLTed files to prevent make rebuilding targets (#55587) This simplifies the `finish_stage` rule. Co-authored-by: Zentrik <[email protected]> * add docstring note about `displaysize` and `IOContext` with `context` (#55510) * LinearAlgebra: replace some hardcoded loop ranges with axes (#56243) These are safer in general, as well as easier to read. Also, narrow the scopes of some `@inbounds` annotations. * inference: fix `[modifyfield!|replacefield!]_tfunc`s (#56310) Currently the following code snippet results in an internal error: ```julia julia> func(x) = @atomic :monotonic x[].count += 1; julia> let;Base.Experimental.@force_compile x = Ref(nothing) func(x) end Internal error: during type inference of ... ``` This issue is caused by the incorrect use of `_fieldtype_tfunc(𝕃, o, f)` within `modifyfield!_tfunc`, specifically because `o` should be `widenconst`ed, but it isn’t. By using `_fieldtype_tfunc` correctly, we can avoid the error through error-catching in `abstract_modifyop!`. This commit also includes a similar fix for `replacefield!_tfunc` as well. * inference: don't allow `SSAValue`s in assignment lhs (#56314) In `InferenceState` the lhs of a `:=` expression should only contain `GlobalRef` or `SlotNumber` and no other IR elements. Currently when `SSAValue` appears in `lhs`, the invalid assignment effect is somehow ignored, but this is incorrect anyway, so this commit removes that check. Since `SSAValue` should not appear in `lhs` in the first place, this is not a significant change though. * Fix `unsafe_read` for `IOBuffer` with non dense data (#55776) Fixes one part of #54636 It was only safe to use the following if `from.data` was a dense vector of bytes. ```julia GC.@preserve from unsafe_copyto!(p, pointer(from.data, from.ptr), adv) ``` This PR adds a fallback suggested by @matthias314 in https://discourse.julialang.org/t/copying-bytes-from-abstractvector-to-ptr/119408/7 * support `isless` for zero-dimensional `AbstractArray`s (#55772) Fixes #55771 * inference: don't add backdge when `applicable` inferred to return `Bool` (#56316) Also just as a minor backedge reduction optimization, this commit avoids adding backedges when `applicable` is inferred to return `::Bool`. * Mark `require_one_based_indexing` and `has_offset_axes` as public (#56196) The discussion here mentions `require_one_based_indexing` being part of the public API: https://github.com/JuliaLang/julia/pull/43263 Both functions are also documented (albeit in the dev docs): * `require_one_based_indexing`: https://docs.julialang.org/en/v1/devdocs/offset-arrays/#man-custom-indices * `has_offset_axes`: https://docs.julialang.org/en/v1/devdocs/offset-arrays/#For-objects-that-mimic-AbstractArray-but-are-not-subtypes Towards https://github.com/JuliaLang/julia/issues/51335. --------- Co-authored-by: Matt Bauman <[email protected]> * Avoid some allocations in various `println` methods (#56308) * Add a developer documentation section to the `LinearAlgebra` docs (#56324) Functions that are meant for package developers may go here, instead of the main section that is primarily for users. * drop require lock when not needed during loading to allow parallel precompile loading (#56291) Fixes `_require_search_from_serialized` to first acquire all start_loading locks (using a deadlock-free batch-locking algorithm) before doing stalechecks and the rest, so that all the global computations happen behind the require_lock, then the rest can happen behind module-specific locks, then (as before) extensions can be loaded in parallel eventually after `require` returns. * Make `String(::Memory)` copy (#54457) A more targeted fix of #54369 than #54372 Preserves the performance improvements added in #53962 by creating a new internal `_unsafe_takestring!(v::Memory{UInt8})` function that does what `String(::Memory{UInt8})` used to do. * 🤖 [master] Bump the Pkg stdlib from 799dc2d54 to 116ba910c (#56336) Stdlib: Pkg URL: https://github.com/JuliaLang/Pkg.jl.git Stdlib branch: master Julia branch: master Old commit: 799dc2d54 New commit: 116ba910c Julia version: 1.12.0-DEV Pkg version: 1.12.0 Bump invoked by: @IanButterworth Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: https://github.com/JuliaLang/Pkg.jl/compare/799dc2d54c4e809b9779de8c604564a5b3befaa0...116ba910c74ab565d348aa8a50d6dd10148f11ab ``` $ git log --oneline 799dc2d54..116ba910c 116ba910c fix Base.unreference_module call (#4057) 6ed1d2f40 do not show right hand progress without colors (#4047) ``` Co-authored-by: Dilum Aluthge <[email protected]> * Wall-time/all tasks profiler (#55889) One limitation of sampling CPU/thread profiles, as is currently done in Julia, is that they primarily capture samples from CPU-intensive tasks. If many tasks are performing IO or contending for concurrency primitives like semaphores, these tasks won’t appear in the profile, as they aren't scheduled on OS threads sampled by the profiler. A wall-time profiler, like the one implemented in this PR, samples tasks regardless of OS thread scheduling. This enables profiling of IO-heavy tasks and detecting areas of heavy contention in the system. Co-developed with @nickrobinson251. * recommend explicit `using Foo: Foo, ...` in package code (was: "using considered harmful") (#42080) I feel we are heading up against a "`using` crisis" where any new feature that is implemented by exporting a new name (either in Base or a package) becomes a breaking change. This is already happening (https://github.com/JuliaGPU/CUDA.jl/pull/1097, https://github.com/JuliaWeb/HTTP.jl/pull/745) and as projects get bigger and more names are exported, the likelihood of this rapidly increases. The flaw in `using Foo` is fundamental in that you cannot lexically see where a name comes from so when two packages export the same name, you are screwed. Any code that relies on `using Foo` and then using an exported name from `Foo` is vulnerable to another dependency exporting the same name. Therefore, I think we should start to strongly discourage the use of `using Foo` and only recommend `using Foo` for ephemeral work (e.g. REPL work). --------- Co-authored-by: Dilum Aluthge <[email protected]> Co-authored-by: Mason Protter <[email protected]> Co-authored-by: Max Horn <[email protected]> Co-authored-by: Matt Bauman <[email protected]> Co-authored-by: Alex Arslan <[email protected]> Co-authored-by: Ian Butterworth <[email protected]> Co-authored-by: Neven Sajko <[email protected]> * Change some hardcoded loop ranges to axes in dense linalg functions (#56348) These should be safer in general, and are also easier to reason about. * Make `LinearAlgebra.haszero` public (#56223) The trait `haszero` is used to check if a type `T` has a unique zero defined using `zero(T)`. This lets us dispatch to optimized paths without losing generality. This PR makes the function public so that this may be extended by packages (such as `StaticArrays`). * remove spurious parens in profiler docs (#56357) * Fix `log_quasitriu` for internal scaling `s=0` (#56311) This PR is a potential fix for #54833. ## Description The function https://github.com/JuliaLang/julia/blob/2a06376c18afd7ec875335070743dcebcd85dee7/stdlib/LinearAlgebra/src/triangular.jl#L2220 computes $\boldsymbol{A}^{\dfrac{1}{2^s}} - \boldsymbol{I}$ for a real-valued $2\times 2$ matrix $\boldsymbol{A}$ using Algorithm 5.1 in [R1]. However, the algorithm in [R1] as well as the above function do not handle the case $s=0.$ This fix extends the function to compute $\boldsymbol{A}^{\dfrac{1}{2^s}} - \boldsymbol{I} \Bigg|_{s=0} = \boldsymbol{A} - \boldsymbol{I}.$ ## Checklist - [X] Fix code: `stdlib\LinearAlgebra\src\triangular.jl` in function `_sqrt_pow_diag_block_2x2!(A, A0, s)`. - [X] Add test case: `stdlib\LinearAlgebra\test\triangular.jl`. - [X] Update `NEWS.md`. - [X] Testing and self review. | Tag | Reference | | --- | --- | | <nobr>[R1]</nobr> | Al-Mohy, Awad H. and Higham, Nicholas J. "Improved Inverse Scaling and Squaring Algorithms for the Matrix Logarithm", 2011, url: https://eprints.maths.manchester.ac.uk/1687/1/paper11.pdf | --------- Co-authored-by: Daniel Karrasch <[email protected]> Co-authored-by: Oscar Smith <[email protected]> * loading: clean up more concurrency issues (#56329) Guarantee that `__init__` runs before `using` returns. Could be slightly breaking for people that do crazy things inside `__init__`, but just don't do that. Since extensions then probably load after `__init__` (or at least, run their `__init__` after), this is a partial step towards changing things so that extensions are guaranteed to load if using all of their triggers before the corresponding `using` returns Fixes #55556 * make `_unsetindex` fast for isbits eltype (#56364) fixes https://github.com/JuliaLang/julia/issues/56359#issuecomment-2441537634 ``` using Plots function f(n) a = Vector{Int}(undef, n) s = time_ns() resize!(a, 8) time_ns() - s end x = 8:10:1000000 y = f.(x) plot(x, y) ``` ![image](https://github.com/user-attachments/assets/5a1fb963-7d44-4cac-bedd-6f0733d4cf56) * improved `eltype` for `flatten` with tuple argument (#55946) We have always had ``` julia> t = (Int16[1,2], Int32[3,4]); eltype(Iterators.flatten(t)) Any ``` With this PR, the result is `Signed` (`promote_typejoin` applied to the element types of the tuple elements). The same applies to `NamedTuple`: ``` julia> nt = (a = [1,2], b = (3,4)); eltype(Iterators.flatten(nt)) Any # old Int64 # new ``` * Reland "Reroute (Upper/Lower)Triangular * Diagonal through __muldiag #55984" (#56270) This relands #55984 which was reverted in #56267. Previously, in #55984, the destination in multiplying triangular matrices with diagonals was also assumed to be triangular, which is not necessarily the case in `mul!`. Tests for this case, however, were being run non-deterministically, so this wasn't caught by the CI runs. This improves performance: ```julia julia> U = UpperTriangular(rand(100,100)); D = Diagonal(rand(size(U,2))); C = similar(U); julia> @btime mul!($C, $D, $U); 1.517 μs (0 allocations: 0 bytes) # nightly 1.116 μs (0 allocations: 0 bytes) # This PR ``` * Add one-arg `norm` method (#56330) This reduces the latency of `norm` calls, as the single-argument method lacks branches and doesn't use aggressive constant propagation, and is therefore simpler to compile. Given that a lot of `norm` calls use `p==2`, it makes sense for us to reduce the latency on this call. ```julia julia> using LinearAlgebra julia> A = rand(2,2); julia> @time norm(A); 0.247515 seconds (390.09 k allocations: 19.993 MiB, 33.57% gc time, 99.99% compilation time) # master 0.067201 seconds (121.24 k allocations: 6.067 MiB, 99.98% compilation time) # this PR ``` An example of an improvement in ttfx because of this: ```julia julia> A = rand(2,2); julia> @time A ≈ A; 0.556475 seconds (1.16 M allocations: 59.949 MiB, 24.14% gc time, 100.00% compilation time) # master 0.333114 seconds (899.85 k allocations: 46.574 MiB, 8.11% gc time, 99.99% compilation time) # this PR ``` * fix a forgotten rename `readuntil` -> `copyuntil` (#56380) Fixes https://github.com/JuliaLang/julia/issues/56352, with the repro in that issue: ``` Master: 1.114874 seconds (13.01 M allocations: 539.592 MiB, 3.80% gc time) After: 0.369492 seconds (12.99 M allocations: 485.031 MiB, 10.73% gc time) 1.10: 0.341114 seconds (8.36 M allocations: 454.242 MiB, 2.69% gc time) ``` * remove unnecessary operations from `typejoin_union_tuple` (#56379) Removes the unnecessary call to `unwrap_unionall` and type assertion. * precompile: fix performance issues with IO (#56370) The string API here rapidly becomes unusably slow if dumping much debug output during precompile. Fix the design here to use an intermediate IO instead to prevent that. * cache the `find_all_in_cache_path` call during parallel precompilation (#56369) Before (in an environment with DifferentialEquations.jl): ```julia julia> @time Pkg.precompile() 0.733576 seconds (3.44 M allocations: 283.676 MiB, 6.24% gc time) julia> isfile_calls[1:10] 10-element Vector{Pair{String, Int64}}: "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Printf/3FQLY_zHycD.ji" => 178 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Printf/3FQLY_xxrt3.ji" => 178 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Dates/p8See_xxrt3.ji" => 158 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Dates/p8See_zHycD.ji" => 158 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/TOML/mjrwE_zHycD.ji" => 155 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/TOML/mjrwE_xxrt3.ji" => 155 "/home/kc/.julia/compiled/v1.12/Preferences/pWSk8_4Qv86.ji" => 152 "/home/kc/.julia/compiled/v1.12/Preferences/pWSk8_juhqb.ji" => 152 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_zHycD.ji" => 144 "/home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_xxrt3.ji" => 144 ``` After: ```julia julia> @time Pkg.precompile() 0.460077 seconds (877.59 k allocations: 108.075 MiB, 4.77% gc time) julia> isfile_calls[1:10] 10-element Vector{Pair{String, Int64}}: "/tmp/jl_a5xFWK/Project.toml" => 15 "/tmp/jl_a5xFWK/Manifest.toml" => 7 "/home/kc/.julia/registries/General.toml" => 6 "/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Markdown/src/Markdown.jl" => 3 "/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl" => 3 "/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/Distributed.jl" => 3 "/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/UUIDs/src/UUIDs.jl" => 3 "/home/kc/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/LibCURL/src/LibCURL.jl" => 3 ``` Performance is improved and we are not calling `isfile` on a bunch of the same ji files hundreds times. Benchmark is made on a linux machine so performance diff should be a lot better on Windows where these `isfile_casesensitive` call is much more expensive. Fixes https://github.com/JuliaLang/julia/issues/56366 --------- Co-authored-by: KristofferC <[email protected]> Co-authored-by: Ian Butterworth <[email protected]> * [docs] Fix note admonition in llvm-passes.md (#56392) At the moment this is rendered incorrectly: https://docs.julialang.org/en/v1.11.1/devdocs/llvm-passes/#JuliaLICM * structure-preserving broadcast for `SymTridiagonal` (#56001) With this PR, certain broadcasting operations preserve the structure of a `SymTridiagonal`: ```julia julia> S = SymTridiagonal([1,2,3,4], [1,2,3]) 4×4 SymTridiagonal{Int64, Vector{Int64}}: 1 1 ⋅ ⋅ 1 2 2 ⋅ ⋅ 2 3 3 ⋅ ⋅ 3 4 julia> S .* 2 4×4 SymTridiagonal{Int64, Vector{Int64}}: 2 2 ⋅ ⋅ 2 4 4 ⋅ ⋅ 4 6 6 ⋅ ⋅ 6 8 ``` This was deliberately disabled on master, but I couldn't find any test that fails if this is enabled. * 🤖 [master] Bump the Pkg stdlib from 116ba910c to 9f8e11a4c (#56386) Stdlib: Pkg URL: https://github.com/JuliaLang/Pkg.jl.git Stdlib branch: master Julia branch: master Old commit: 116ba910c New commit: 9f8e11a4c Julia version: 1.12.0-DEV Pkg version: 1.12.0 Bump invoked by: @IanButterworth Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: https://github.com/JuliaLang/Pkg.jl/compare/116ba910c74ab565d348aa8a50d6dd10148f11ab...9f8e11a4c0efb3b68a1e25a33f372f398c89cd66 ``` $ git log --oneline 116ba910c..9f8e11a4c 9f8e11a4c strip out tree_hash for stdlibs that have have been freed in newer julia versions (#4062) c0df25a47 rm dead code (#4061) ``` Co-authored-by: Dilum Aluthge <[email protected]> * load extensions with fewer triggers earlier (#49891) Aimed to support the use case in https://github.com/JuliaLang/julia/issues/48734#issuecomment-1554626135. https://github.com/KristofferC/ExtSquared.jl is an example, see specifically https://github.com/KristofferC/ExtSquared.jl/blob/ded7c57d6f799674e3310b8174dfb07591bbe025/ext/BExt.jl#L4. I think this makes sense, happy for a second pair of eyes though. cc @termi-official --------- Co-authored-by: KristofferC <[email protected]> Co-authored-by: Cody Tapscott <[email protected]> * Dispatch in generic_matmatmul (#56384) Replacing the branches by dispatch reduces latency, presumably because there's less dead code in the method. ```julia julia> using LinearAlgebra julia> A = rand(Int,2,2); B = copy(A); C = similar(A); julia> @time mul!(C, A, B, 1, 2); 0.363944 seconds (1.65 M allocations: 84.584 MiB, 37.57% gc time, 99.99% compilation time) # master 0.102676 seconds (176.55 k allocations: 8.904 MiB, 27.04% gc time, 99.97% compilation time) # this PR ``` The latency is now distributed between the different branches: ```julia julia> @time mul!(C, A, B, 1, 2); 0.072441 seconds (176.55 k allocations: 8.903 MiB, 99.97% compilation time) julia> @time mul!(C, A', B, 1, 2); 0.085817 seconds (116.44 k allocations: 5.913 MiB, 99.96% compilation time: 4% of which was recompilation) julia> @time mul!(C, A', B', 1, 2); 0.345337 seconds (1.07 M allocations: 54.773 MiB, 25.77% gc time, 99.99% compilation time: 40% of which was recompilation) ``` It would be good to look into why there's recompilation in the last case, but the branch is less commonly taken than the others that have significantly lower latency after this PR. * Add `atol` to addmul tests (#56210) This avoids the issues as in https://github.com/JuliaLang/julia/issues/55781 and https://github.com/JuliaLang/julia/issues/55779 where we compare small numbers using a relative tolerance. Also, in this PR, I have added an extra test, so now we compare both `A * B * alpha + C * beta` and `A * B * alpha - C * beta` with the corresponding in-place versions. The idea is that if the terms `A * B * alpha` and ` C * beta` have similar magnitudes, at least one of the two expressions will usually result in a large enough number that may be compared using a relative tolerance. I am unsure if the `atol` chosen here is optimal, as I have ballparked it to use the maximum `eps` by looking at all the `eltype`s involved. Fixes #55781 Fixes #55779 * Export jl_gc_new_weakref again via julia.h (#56373) This is how it used for at least Julia 1.0 - 1.11 Closes #56367 * InteractiveUtils: define `InteractiveUtils.@code_ircode` (#56390) * Fix some missing write barriers and add some helpful comments (#56396) I was trying some performance optimization which didn't end up working out, but in the process I found two missing write barriers and added some helpful comments for future readers, so that part is probably still useful. * compiler: fix specialization mistake introduced by #40985 (#56404) Hopefully there aren't any others like this hiding around? Not useful to make a new closure for every method that we inline, since we just called `===` inside it * Avoid racy double-load of binding restriction in `import_module` (#56395) Fixes #56333 * define `InteractiveUtils.@infer_[return|exception]_type` (#56398) Also simplifies the definitions of `@code_typed` and the other similar macros. * irinterp: set `IR_FLAG_REFINED` for narrowed `PhiNode`s (#56391) `adce_pass!` can transform a `Union`-type `PhiNode` into a narrower `PhiNode`, but in such cases, the `IR_FLAG_REFINED` flag isn’t set on that `PhiNode` statement. By setting this flag, irinterp can perform statement reprocessing using the narrowed `PhiNode`, enabling type stability in cases like JuliaLang/julia#56387. - fixes JuliaLang/julia#56387 * document isopen(::Channel) (#56376) This PR has two purposes -- 1) Add some documentation for public API 2) Add a small note about a footgun I've hit a few times: `!isopen(ch)` does not mean that you are "done" with the channel because buffered channels can still have items left in them that need to be taken. --------- Co-authored-by: CY Han <[email protected]> * Make build system respect `FORCE_COLOR` and `NO_COLOR` settings (#56346) Follow up to #53742, but for the build system. CC: @omus. * Add `edges` vector to CodeInstance/CodeInfo to keep backedges as edges (#54894) Appears to add about 11MB (128MB to 139MB) to the system image, and to decrease the stdlib size by 55 MB (325MB to 270MB), so seems overall favorable right now. The edges are computed following the encoding <https://hackmd.io/sjPig55kS4a5XNWC6HmKSg?both#Edges-Encoding> to correctly reflect the backedges. Co-authored-by: Shuhei Kadowaki <[email protected]> * docs: remove `dirname.c` from THIRDPARTY file (#56413) - `dirname.c` was removed by https://github.com/JuliaLang/julia/commit/c2cec7ad57102e4fbb733b8fb79d617a9524f0ae * Allow ext → ext dependency if triggers are a strict superset (#56368) (#56402) Forward port of #56368 - this was a pretty clean port, so it should be good to go once tests pass. * [docs] Fix rendering of warning admonition in llvm passes page (#56412) Follow up to #56392: also the warning in https://docs.julialang.org/en/v1.11.1/devdocs/llvm-passes/#Multiversioning is rendered incorrectly because of a missing space. * Fix dispatch for `rdiv!` with `LU` (#55764) * Remove overwritten method of OffsetArray (#56414) This is overwritten three definitions later in `Base.reshape(A::OffsetArray, inds::Colon)`. Should remove warnings I saw when testing a package that uses it. * Add a missing GC root in constant declaration (#56408) As pointed out in https://github.com/JuliaLang/julia/pull/56224#discussion_r1816974147. * Teach compiler about partitioned bindings (#56299) This commit teaches to compiler to update its world bounds whenever it looks at a binding partition, making the compiler sound in the presence of a partitioned binding. The key adjustment is that the compiler is no longer allowed to directly query the binding table without recording the world bounds, so all the various abstract evaluations that look at bindings need to be adjusted and are no longer pure tfuncs. We used to look at bindings a lot more, but thanks to earlier prep work to remove unnecessary binding-dependent code (#55288, #55289 and #55271), these changes become relatively straightforward. Note that as before, we do not create any binding partitions by default, so this commit is mostly preperatory. --------- Co-authored-by: Shuhei Kadowaki <[email protected]> * Restore JL_NOTSAFEPOINT in jl_stderr_obj (#56407) This is not a function we're really using, but it's used in the embedding examples, so I'm sure somebody would complain if I deleted it or made it a safepoint, so let's just give the same best-effort result as before. * reland "Inlining: Remove outdated code path for GlobalRef movement (#46880)" (#56382) From the description of the original PR: > We used to not allow `GlobalRef` in `PhiNode` at all (because they > could have side effects). However, we then change the IR to make > side-effecting `GlobalRef`s illegal in statement position in general, > so now `PhiNode`s values are just regular value position, so there's > no reason any more to try to move `GlobalRef`s out to statement > position in inlining. Moreover, doing so introduces a bunch of > unnecessary `GlobalRef`s that weren't being moved back. We could fix > that separately by setting appropriate flags, but it's simpler to just > get rid of this special case entirely. This change itself does not sound to have any issues, and in fact, it is very useful for keeping the IR slim, especially in code generated by Cassette-like systems, so I would like to reland it. However, the original PR was reverted in JuliaLang/julia#46951 due to bugs like JuliaLang/julia#46940 and JuliaLang/julia#46943. I could not reproduce these bugs on my end (maybe they have been fixed on some GC-side fixes?), so I believe relanding the original PR’s changes would not cause any issues, but it is necessary to confirm that similar problems do not arise before merging this PR. * copy effects key to `Base.infer_effects` (#56363) Copied from the docstring of `Core.Compiler.Effects`, this makes it easier to figure out what the output of `Base.infer_effects` is actually telling you. * Fix `make install` for asan build (#56347) Now the makescript finds libclang_rt.asan-x86_64.so for example. The change from `-0` to `-1` is as with `-1`, `libclang_rt.asan-*` is searched for in `usr/lib/julia` instead of `usr/lib`. * Add dims check to triangular mul (#56393) This adds a dimension check to triangular matrix multiplication methods. While such checks already exist in the individual branches (occasionally within `BLAS` methods), having these earlier would permit certain optimizations, as we are assured that the axes are compatible. This potentially duplicates the checks, but this is unlikely to be a concern given how cheap the checks are. I've also reused the `check_A_mul_B!_sizes` function that is defined in `bidiag.jl`, instead of hard-coding the checks. Further, I've replaced some hard-coded loop ranges by the corresponding `axes` and `first/lastindex` calls. These are identical under the 1-based indexing assumption, but the `axes` variants are easier to read and reason about. * clarify short-circuit && and || docs (#56420) This clarifies the docs to explain that `a && b` is equivalent to `a ? b : false` and that `a || b` is equivalent to `a ? true : b`. In particular, this explains why the second argument does not need to be a boolean value, which is a common point of confusion. (See e.g. [this discourse thread](https://discourse.julialang.org/t/internals-of-assignment-when-doing-short-circuit-evaluation/122178/2?u=stevengj).) * docs: replace 'leaf types' with 'concrete types' (#56418) Fixes #55044 --------- Co-authored-by: inkydragon <[email protected]> * Remove aggressive constprop annotation on generic_matmatmul_wrapper! (#56400) This annotation seems unnecessary, as the method gets inlined and there's no computation being carried out using the value of the constant. * Clarify the FieldError docstring (#55222) * Allow `Time`s to be rounded to `Period`s (#52629) Co-authored-by: CyHan <[email protected]> Co-authored-by: Curtis Vogt <[email protected]> * Replace unconditional store with cmpswap to avoid deadlocking in jl_fptr_wait_for_compiled_addr (#56444) That unconditional store could overwrite the actual compiled code in that pointer, so make it a cmpswap * Correct nothrow modeling of `get_binding_type` (#56430) As pointed out in https://github.com/JuliaLang/julia/pull/56299#discussion_r1826509185, although the bug predates that PR. * add tip for module docstrings before load (#56445) * compiler: Strengthen some assertions and fix a couple small bugs (#56449) * inference: minor follow-ups to JuliaLang/julia#56299 (#56450) * Ensure that String(::Memory) returns only a String, not any owner (#56438) Fixes #56435 * Take safepoint lock before going to sleep in the scheduler. (#56443) This avoids a deadlock during exit. Between a thread going to sleep and the thread exiting. * Profile: mention `kill -s SIGUSR1 julia_pid` for Linux (#56441) currentlu this route is mentioned in docs https://docs.julialang.org/en/v1/stdlib/Profile/#Triggered-During-Execution but missing from the module docstring, this should help users who have little idea how to "send a kernel signal to a process" to get started --------- Co-authored-by: Ian Butterworth <[email protected]> * Fix and test an overflow issue in `searchsorted` (#56464) And remove `searchsorted` special cases for offset arrays in tests that had the impact of bypassing actually testing `searchsorted` behavior on offset arrays To be clear, after this bugfix the function is still broken, just a little bit less so. * Update docs of calling convention arg in `:foreigncall` AST node (#56417) * `step(::AbstractUnitRange{Bool})` should return `Bool` (#56405) The issue was introduced by #27302 , as ```julia julia> true-false 1 ``` By definitions below, `AbstractUnitRange{Bool} <: OrdinalRange{Bool, Bool}` whose step type is `Bool`. https://github.com/JuliaLang/julia/blob/da74ef1933b12410b217748e0f7fbcbe52e10d29/base/range.jl#L280-L299 --------- Co-authored-by: Matt Bauman <[email protected]> Co-authored-by: Matt Bauman <[email protected]> * fixup! JuliaLang/julia#56028, fix up the type-level escapability check In JuliaLang/julia#56028, the type-level escapability check was changed to use `is_mutation_free_argtype`, but this was a mistake because EA no longer runs for structs like `mutable struct ForeignBuffer{T}; const ptr::Ptr{T}; end`. This commit changes it to use `is_identity_free_argtype` instead, which can be used to detect whether a type may contain any mutable allocations or not. * add `show(::IO, ::ArgEscapeInfo)` * EA: disable finalizer inlining for allocations that are edges of `PhiNode`s (#56455) The current EA-based finalizer inlining implementation can create invalid IR when the target object is later aliased as a `PhiNode`, which was causing #56422. In such cases, finalizer inlining for the allocations that are edges of each `PhiNode` should be avoided, and instead, finalizer inlining should ideally be applied to the `PhiNode` itself, but implementing that is somewhat complex. As a temporary fix, this commit disables inlining in those cases. - fixes #56422 * make `verify_ir` error messages more informative (#56452) Currently, when `verify_ir` finds an error, the `IRCode` is printed, but it's not easy to determine which method instance generated that `IRCode`. This commit adds method instance and code location information to the error message, making it easier to identify the problematic code. E.g.: ```julia [...] 610 │ %95 = builtin Core.tuple(%48, %94)::Tuple{GMT.Gdal.IGeometry, GMT.Gdal.IGeometry} └─── return %95 ERROR: IR verification failed. Code location: ~/julia/packages/GMT/src/gdal_extensions.jl:606 Method instance: MethodInstance for GMT.Gdal.helper_2geoms(::Matrix{Float64}, ::Matrix{Float64}) Stacktrace: [1] error(::String, ::String, ::String, ::Symbol, ::String, ::Int32, ::String, ::String, ::Core.MethodInstance) @ Core.Compiler ./error.jl:53 [...] ``` * [GHA] Explicitly install Julia for whitespace workflow (#56468) So far we relied on the fact that Julia comes in the default Ubuntu images on GitHub Actions runners, but this may change in the future (although there's apparently no plan in this direction for the time being). To make the workflow more future-proof, we now explicitly install Julia using a dedicated workflow. * Allow taking Matrix slices without an extra allocation (#56236) Since changing Array to use Memory as the backing, we had the option of making non-Vector arrays more flexible, but had instead preserved the restriction that they must be zero offset and equal in length to the Memory. This results in extra complexity, restrictions, and allocations however, but doesn't gain many known benefits. Nanosoldier shows a decrease in performance on linear eachindex loops, which we theorize is due to a minor failure to CSE before SCEV or a lack of NUW/NSW on the length multiplication calculation. * [late-gc-lowering] null-out GC frame slots for dead objects (#52935) Should fix https://github.com/JuliaLang/julia/issues/51818. MWE: ```julia function testme() X = @noinline rand(1_000_000_00) Y = @noinline sum(X) X = nothing GC.gc() return Y end ``` Note that it now stores a `NULL` in the GC frame before calling `jl_gc_collect`. Before: ```llvm ; Function Signature: testme() ; @ /Users/dnetto/Personal/test.jl:3 within `testme` define double @julia_testme_535() #0 { top: %gcframe1 = alloca [3 x ptr], align 16 call void @llvm.memset.p0.i64(ptr align 16 %gcframe1, i8 0, i64 24, i1 true) %pgcstack = call ptr inttoptr (i64 6595051180 to ptr)(i64 262) #10 store i64 4, ptr %gcframe1, align 16 %task.gcstack = load ptr, ptr %pgcstack, align 8 %frame.prev = getelementptr inbounds ptr, ptr %gcframe1, i64 1 store ptr %task.gcstack, ptr %frame.prev, align 8 store ptr %gcframe1, ptr %pgcstack, align 8 ; @ /Users/dnetto/Personal/test.jl:4 within `testme` %0 = call nonnull ptr @j_rand_539(i64 signext 100000000) %gc_slot_addr_0 = getelementptr inbounds ptr, ptr %gcframe1, i64 2 store ptr %0, ptr %gc_slot_addr_0, align 16 ; @ /Users/dnetto/Personal/test.jl:5 within `testme` %1 = call double @j_sum_541(ptr nonnull %0) ; @ /Users/dnetto/Personal/test.jl:7 within `testme` ; ┌ @ gcutils.jl:132 within `gc` @ gcutils.jl:132 call void @jlplt_ijl_gc_collect_543_got.jit(i32 1) %frame.prev4 = load ptr, ptr %frame.prev, align 8 store ptr %frame.prev4, ptr %pgcstack, align 8 ; └ ; @ /Users/dnetto/Personal/test.jl:8 within `testme` ret double %1 } ``` After: ```llvm ; Function Signature: testme() ; @ /Users/dnetto/Personal/test.jl:3 within `testme` define double @julia_testme_752() #0 { top: %gcframe1 = alloca [3 x ptr], align 16 call void @llvm.memset.p0.i64(ptr align 16 %gcframe1, i8 0, i64 24, i1 true) %pgcstack = call ptr inttoptr (i64 6595051180 to ptr)(i64 262) #10 store i64 4, ptr %gcframe1, align 16 %task.gcstack = load ptr, ptr %pgcstack, align 8 %frame.prev = getelementptr inbounds ptr, ptr %gcframe1, i64 1 store ptr %task.gcstack, ptr %frame.prev, align 8 store ptr %gcframe1, ptr %pgcstack, align 8 ; @ /Users/dnetto/Personal/test.jl:4 within `testme` %0 = call nonnull ptr @j_rand_756(i64 signext 100000000) %gc_slot_addr_0 = getelementptr inbounds ptr, ptr %gcframe1, i64 2 store ptr %0, ptr %gc_slot_addr_0, align 16 ; @ /Users/dnetto/Personal/test.jl:5 within `testme` %1 = call double @j_sum_758(ptr nonnull %0) store ptr null, ptr %gc_slot_addr_0, align 16 ; @ /Users/dnetto/Personal/test.jl:7 within `testme` ; ┌ @ gcutils.jl:132 within `gc` @ gcutils.jl:132 call void @jlplt_ijl_gc_collect_760_got.jit(i32 1) %frame.prev6 = load ptr, ptr %frame.prev, align 8 store ptr %frame.prev6, ptr %pgcstack, align 8 ; └ ; @ /Users/dnetto/Personal/test.jl:8 within `testme` ret double %1 } ``` * Added test for resolving array references in exprresolve (#56471) added test to take care of non-real-index handling while resolving array references in exprresolve to test julia/base/cartesian.jl - line 427 to 432 * Fix and test searchsorted for arrays whose first index is `typemin(Int)` (#56474) This fixes the issue reported in https://github.com/JuliaLang/julia/issues/56457#issuecomment-2457223264 which, combined with #56464 which fixed the issue in the OP, fixes #56457. `searchsortedfirst` was fine all along, but I added it to tests regardless. * Move Core.Compiler into Base This is the first step in what I am hoping will eventually result in making the compiler itself and upgradable stdlib. Over time, we've gained several non-Base consumers of `Core.Compiler`, and we've reached a bit of a breaking point where maintaining those downstream dependencies is getting more difficult than the close coupling of Core.Compiler to the runtime is worth. In this first step, I am moving Core.Compiler into Base, ending the duplication of common data structure and generic functions between Core.Compiler and Base. This split goes back quite far (although not all the way) to the early days of Julia and predates the world-age mechanism. The extant Base and Core.Compiler environments have some differences (other than the duplication). I think the primary ones are (but I will add more here if somebody points one out). - `Core.Compiler` does not use `getproperty` - `Core.Compiler` does not have extensible `==` equality In this, I decided to retain the former by setting `getproperty = getfield` for Core.Compiler itself (though of course not for the datatstructures shared with Base). I don't think it's strictly necessary, but might as well. For equality, I decided the easiest thing to do would be to try to merge the equalities and see what happens. In general, Core.Compiler is relatively restricted in the kinds of equality comparisons it can make, so I think it'll work out fine, but we can revisit this. This seems to be fully working and most of this is just moving code around. I think most of that refactoring is independently useful, so I'll pull some of it out into separate PRs to make this PR more manageable. * Delete buggy `stat(::Integer)` method (#54855) "Where did someone get a RawFD as an integer anyway?" -@stefankarpinski See also #51711 Fixes #51710 * missing gc-root store in subtype (#56472) Fixes #56141 Introduced by #52228 (a624d445c02c) * further defer jl_insert_backedges after loading (#56447) Finish fully breaking the dependency between method insertions and inferring whether the cache is valid. The cache should be inferable in parallel and in aggregate after all loading is finished. This prepares us for moving this code into Julia (Core.Compiler) next. * count bytes allocated through malloc more precisely (#55223) Should make the accounting for memory allocated through malloc a bit more accurate. Should also simplify the accounting code by eliminating the use of `jl_gc_count_freed` in `jl_genericmemory_to_string`. * Fix external IO loop thead interaction and add function to Base.Experimental to facilitate it's use. Also add a test. (#55529) While looking at https://github.com/JuliaLang/julia/issues/55525 I found that the implementation wasn't working correctly. I added it to Base.Experimental so people don't need to handroll their own and am also testing a version of what the issue was hitting. * [REPL] raise default implicit `show` limit to 1MiB (#56297) https://github.com/JuliaLang/julia/pull/53959#issuecomment-2426946640 I would like to understand more where these issues are coming from; it would be easy to exempt some types from Base or Core with ```julia REPL.show_limited(io::IO, mime::MIME, x::SomeType) = show(io, mime, x) ``` but I'm not sure which are causing problems in practice. But meanwhile I think raising the limit makes sense. * Add a docstring for `Base.divgcd` (#53769) Co-authored-by: Sukera <[email protected]> * Fix compilation warning on aarch64-linux (#56480) This fixes the warning: ``` /cache/build/default-aws-aarch64-ci-1-3/julialang/julia-master/src/stackwalk.c: In function 'jl_simulate_longjmp': /cache/build/default-aws-aarch64-ci-1-3/julialang/julia-master/src/stackwalk.c:995:22: warning: initialization of 'mcontext_t *' {aka 'struct sigcontext *'} from incompatible pointer type 'struct unw_sigcontext *' [-Wincompatible-pointer-types] 995 | mcontext_t *mc = &c->uc_mcontext; | ^ ``` This is the last remaining warning during compilation on aarch64-linux. * Make Compiler an independent package This is a further extension to #56128 to make the compiler into a proper independent, useable outside of `Base` as `using Compiler` in the same way that `JuliaSyntax` works already. InteractiveUtils gains a new `@activate` macro that can be used to activate an outside Compiler package, either for reflection only or for codegen also. * Make heap size hint available as an env variable (#55631) This makes `JULIA_HEAP_SIZE_HINT` the environment variable version of the `--heap-size-hint` command-line flag. Seems like there was interest in https://github.com/JuliaLang/julia/pull/45369#issuecomment-1544204022. The same syntax is used as for the command-line version with, for example, `2G` => 2 GB and `200M` => 200 MB. @oscardssmith want to take a look? * Allow indexing `UniformScaling` with `CartesianIndex{2}` (#56461) Since indexing with two `Integer`s is defined, we might as well define indexing with a `CartesianIndex`. This makes certain loops convenient where the index is obtained using `eachindex`. * Simplify first index in `FastContiguousSubArray` definition (#56491) Since `Slice <: AbstractUnitRange` and `Union{Slice, AbstractUnitRange} == AbstractUnitRange`, we may simplify the first index. * Make `popat!` support `@inbounds` (#56323) Co-authored-by: Jishnu Bhattacharya <[email protected]> * NEWS.md: clarify `--trim` (#56460) Co-authored-by: Matt Bauman <[email protected]> * Remove aggressive constprop annotation from 2x2 and 3x3 matmul (#56453) Removing these annotations reduces ttfx slightly. ```julia julia> using LinearAlgebra julia> A = rand(2,2); julia> @time mul!(similar(A), A, A, 1, 2); 0.296096 seconds (903.49 k allocations: 44.313 MiB, 4.25% gc time, 99.98% compilation time) # nightly 0.286009 seconds (835.88 k allocations: 40.732 MiB, 3.29% gc time, 99.98% compilation time) # this PR ``` * `sincos` for non-float symmetric matrices (#56484) Ensures that the `eltype` of the array to which the result of `sincos` is a floating-point one, even if the argument doesn't have a floating-point `eltype`. After this, the following works: ```julia julia> A = diagm(0=>1:3) 3×3 Matrix{Int64}: 1 0 0 0 2 0 0 0 3 julia> sincos(A) ([0.8414709848078965 0.0 0.0; 0.0 0.9092974268256817 0.0; 0.0 0.0 0.1411200080598672], [0.5403023058681398 0.0 0.0; 0.0 -0.4161468365471424 0.0; 0.0 0.0 -0.9899924966004454]) ``` * Specialize 2-arg `show` for `LinearIndices` (#56482) After this, ```julia julia> l = LinearIndices((1:3, 1:4)); julia> show(l) LinearIndices((1:3, 1:4)) ``` The printed form is a valid constructor. * Avoid constprop in `syevd!` and `syev!` (#56442) This improves compilation times slightly: ```julia julia> using LinearAlgebra julia> A = rand(2,2); julia> @time eigen!(Hermitian(A)); 0.163380 seconds (180.51 k allocations: 8.760 MiB, 99.88% compilation time) # master 0.155285 seconds (163.77 k allocations: 7.971 MiB, 99.87% compilation time) # This PR ``` The idea is that the constant propagation is only required to infer the return type, and isn't necessary in the body of the method. We may therefore annotate the body with a `@constprop :none`. * make: define `basecompiler.ji` target (#56498) For easier experimentation with just the bootstrap process. Additionally, as a follow-up to JuliaLang/julia#56409, this commit also includes some minor cosmetic changes. * speed up bootstrapping by compiling few optimizer subroutines earlier (#56501) Speeds up the bootstrapping process by about 30 seconds. * remove top-level branches checking for Base (#56507) These are no longer needed, now that the files are no longer included twice. * Undo the decision to publish incomplete types to the binding table (#56497) This effectively reverts #36121 and replaces it with #36111, which was the originally proposed alternative to fix #36104. To recap, the question is what should happen for ``` module Foo struct F v::Foo.F end end ``` i.e. where the type reference tries to refer to the newly defined type via its global path. In #36121 we adjusted things so that we first assign the type to its global binding and then evaluate the field type (leaving the type in an incomplete state in the meantime). The primary reason that this choice was that we would have to deal with incomplete types assigned to global bindings anyway if we ever did #32658. However, I think this was the wrong choice. There is a difference between allowing incomplete types and semantically forcing incomplete types to be globally observable every time a new type is defined. The situation was a little different four years ago, but with more extensive threading (which can observe the incompletely constructed type) and the upcoming completion of bindings partition, the situation is different. For bindings partition in particular, this would require two invalidations on re-definition, one to the new incomplete type and then back to the complete type. I don't think this is worth it, for the (somewhat niche and possibly-should-be- deprecated-future) case of refering to incompletely defined types by their global names. So let's instead try the hack in #36111, which does a frontend rewrite of the global path. This should be sufficient to at least address the obvious cases. * Merge identical methods for Symmetric/Hermitian and SymTridiagonal (#56434) Since the methods do identical things, we may define each method once for a union of types instead of defining methods for each type. * Specialize findlast for integer AbstractUnitRanges and StepRanges (#54902) For monotonic ranges, `findfirst` and `findlast` with `==(val)` as the predicate should be identical, as each value appears only once in the range. Since `findfirst` is specialized for some ranges, we may define `findlast` as well analogously. On v"1.12.0-DEV.770" ```julia julia> @btime findlast(==(1), $(Ref(1:1_000))[]) 1.186 μs (0 allocations: 0 bytes) 1 ``` This PR ```julia julia> @btime findlast(==(1), $(Ref(1:1_000))[]) 3.171 ns (0 allocations: 0 bytes) 1 ``` I've also specialized `findfirst(iszero, r::AbstractRange)` to make this be equivalent to `findfirst(==(0), ::AbstractRange)` for numerical ranges. Similarly, for `isone`. These now take the fast path as well. Thirdly, I've added some `convert` calls to address issues like ```julia julia> r = Int128(1):Int128(1):Int128(4); julia> findfirst(==(Int128(2)), r) |> typeof Int128 julia> keytype(r) Int64 ``` This PR ensures that the return type always corresponds to `keytype`, which is what the docstring promises. This PR also fixes ```julia julia> findfirst(==(0), UnitRange(-0.5, 0.5)) ERROR: InexactError: Int64(0.5) Stacktrace: [1] Int64 @ ./float.jl:994 [inlined] [2] findfirst(p::Base.Fix2{typeof(==), Int64}, r::UnitRange{Float64}) @ Base ./array.jl:2397 [3] top-level scope @ REPL[1]:1 ``` which now returns `nothing`, as expected. * Loop over `Iterators.rest` in `_foldl_impl` (#56492) For reasons that I don't understand, this improves performance in `mapreduce` in the following example: ```julia julia> function g(A) for col in axes(A,2) mapreduce(iszero, &, view(A, UnitRange(axes(A,1)), col), init=true) || return false end return true end g (generic function with 2 methods) julia> A = zeros(2, 10000); julia> @btime g($A); 28.021 μs (0 allocations: 0 bytes) # nightly v"1.12.0-DEV.1571" 12.462 μs (0 allocations: 0 bytes) # this PR julia> A = zeros(1000,1000); julia> @btime g($A); 372.080 μs (0 allocations: 0 bytes) # nightly 321.753 μs (0 allocations: 0 bytes) # this PR ``` It would be good to understand what the underlying issue is, as the two seem equivalent to me. Perhaps this form makes it clear that it's not, in fact, an infinite loop? * better error message for rpad/lpad with zero-width padding (#56488) Closes #45339 — throw a more informative `ArgumentError` message from `rpad` and `lpad` if a zero-`textwidth` padding is passed (not a `DivideError`). If the padding character has `ncodeunits == 1`, suggests that maybe they want `str * pad^max(0, npad - ncodeunits(str))` instead. * Safer indexing in dense linalg methods (#56451) Ensure that `eachindex` is used consistently alongside `@inbounds`, and use `diagind` to obtain indices along a diagonal. * The `info` in LAPACK calls should be a Ref instead of a Ptr (#56511) Co-authored-by: Viral B. Shah <[email protected]> * Scaling loop instead of broadcasting in strided matrix exp (#56463) Firstly, this is easier to read. Secondly, this merges the two loops into one. Thirdly, this avoids the broadcasting latency. ```julia julia> using LinearAlgebra julia> A = rand(2,2); julia> @time LinearAlgebra.exp!(A); 0.952597 seconds (2.35 M allocations: 116.574 MiB, 2.67% gc time, 99.01% compilation time) # master 0.877404 seconds (2.17 M allocations: 106.293 MiB, 2.65% gc time, 99.99% compilation time) # this PR ``` The performance also improves as there are fewer allocations in the first branch (`opnorm(A, 1) <= 2.1`): ```julia julia> B = diagm(0=>im.*(float.(1:200))./200, 1=>(1:199)./400, -1=>(1:199)./400); julia> opnorm(B,1) 1.9875 julia> @btime exp($B); 5.066 ms (30 allocations: 4.89 MiB) # nightly v"1.12.0-DEV.1581" 4.926 ms (27 allocations: 4.28 MiB) # this PR ``` * codegen: Respect binding partition (#56494) Minor changes to make codegen correct in the face of partitioned constant bindings. Does not yet handle the envisioned semantics for globals that change restriction type, which will require a fair bit of additional work. * Profile: fix Compiler short path (#56515) * Check `isdiag` in dense trig functions (#56483) This improves performance for dense diagonal matrices, as we may apply the function only to the diagonal elements. ```julia julia> A = diagm(0=>rand(100)); julia> @btime cos($A); 349.211 μs (22 allocations: 401.58 KiB) # nightly v"1.12.0-DEV.1571" 16.215 μs (7 allocations: 80.02 KiB) # this PR ``` --------- Co-authored-by: Daniel Karrasch <[email protected]> * Profile: add helper method for printing profile report to file (#56505) The IOContext part is isn't obvious, because otherwise the IO is assumed to be 80 chars wide, which makes for bad reports. * Change in-place exp to out-of-place in matrix trig functions (#56242) This makes the functions work for arbitrary matrix types that support `exp`, but not necessarily the in-place `exp!`. For example, the following works after this: ```julia julia> m = SMatrix{2,2}(1:4); julia> cos(m) 2×2 SMatrix{2, 2, Float64, 4} with indices SOneTo(2)×SOneTo(2): 0.855423 -0.166315 -0.110876 0.689109 ``` There's a slight performance improvement as well because we don't compute `im*A` and `-im*A` separately, but we negate the first to obtain the second. ```julia julia> A = rand(ComplexF64,100,100); julia> @btime sin($A); 2.796 ms (48 allocations: 1.84 MiB) # nightly v"1.12.0-DEV.1571" 2.304 ms (48 allocations: 1.84 MiB) # this PR ``` * Test: Don't change scope kind in `test_{warn,nowarn}` (#56524) This was part of #56509, but is an independent bugfix. The basic issue is that these macro were using `do` block internally. This is undesirable for test macros, because we would like them not to affect the behavior of what they're testing. E.g. right now: ``` julia> using Test julia> const x = 1 1 julia> @test_nowarn const x = 1 ERROR: syntax: `global const` declaration not allowed inside function around /home/keno/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:927 Stacktrace: [1] top-level scope @ REPL[3]:1 ``` This PR just writes out the try/finally manually, so the above works fine after this PR. * For loop instead of while in generic `copyto!` (#56517) This appears to improve performance. ```julia j…

While doing some work on analyzing what code runs at toplevel, I found a few things to simplify or fix: - simplify float.jl loop not to call functions just defined to get back the value just stored there - add method to the correct checkbounds function (instead of a local) - missing world push/pop around jl_interpret_toplevel_thunk after #56509 - jl_lineno use maybe missed in #53799 - add some debugging dump for scm_to_julia_ mistakes

aviatesk reviewed Nov 9, 2024

View reviewed changes

Keno force-pushed the kf/explicitageinc branch from d085375 to 4330bad Compare November 10, 2024 03:02

Keno added a commit that referenced this pull request Nov 11, 2024

Add lowering and interpreter support for :worldinc

47f0d61

Split out from #56509 to facilitate adjusting downstream packages.

Keno mentioned this pull request Nov 11, 2024

Add lowering and interpreter support for :latestworld #56523

Merged

Keno mentioned this pull request Nov 11, 2024

Test: Don't change scope kind in test_{warn,nowarn} #56524

Merged

Keno added a commit that referenced this pull request Nov 12, 2024

Add lowering and interpreter support for :latestworld (#56523)

505907b

Split out from #56509 to facilitate adjusting downstream packages.

vtjnash reviewed Nov 13, 2024

View reviewed changes

Keno mentioned this pull request Nov 14, 2024

Handle :latestworld expr JuliaDebug/JuliaInterpreter.jl#653

Merged

Keno force-pushed the kf/explicitageinc branch from 4330bad to 3bbfc51 Compare November 14, 2024 05:53

Keno mentioned this pull request Nov 15, 2024

Adjust to upcoming change in world age semantics timholy/Revise.jl#864

Merged

Keno added a commit to Keno/SparseArrays.jl that referenced this pull request Nov 18, 2024

Make allowscalar a macro with auto-world-age-increment

0c1204a

Goes along with JuliaLang/julia#56509

Keno mentioned this pull request Nov 18, 2024

Make allowscalar a macro with auto-world-age-increment JuliaSparse/SparseArrays.jl#583

Merged