22 seconds to 3 and now more: Let's fix all of the DifferentialEquations.jl + universe compile times!

[ChrisRackauckas]

Take awhile to precompile with style all of the DiffEq stack, no denial

Our goal is to get the entire DiffEq stack compile times down. From now on we should treat compile time issues and compile time regressions just like we treat performance issues, 😢 😿 😭 and then fix them. For a very long time we did not do this because, unlike runtime performance, we did not have a good way to diagnose the causes, so it was 🤷 whatever we got is what we got, move on. But now, thanks to @timholy, we have the right tools and have learned how to handle compile time issues in a very deep way so the game is on.

Our goal is to get compile times of all standard workflows to at least 0.1 seconds. That is quick enough that you wouldn't care all that much in interactive usage, but still not an unreasonable goal given the benchmarks. This issue is how to get there and how the community is can help. This will cover:

• What have we done so far (so you can learn from our experience)
• How much has it mattered and to what use cases
• How can a user give helpful compile time issues
• What are some helpful tricks and knowledge to share?
• What are some of the next steps

Let's dig in.

What were our first strides?

We have already made a great leap forward in the last week+ since the JuliaCon hackathon. The very long tl;dr i in SciML/DiffEqBase.jl#698 . However, that doesn't capture the full scope of what was actually done:

• Improve precompilation OrdinaryDiffEq.jl#1460
• Reduce the first time to solve from 5 seconds to 1 second for Tsit5 OrdinaryDiffEq.jl#1465
• ftts with a few implicit methods OrdinaryDiffEq.jl#1467
• make the life of inference easier OrdinaryDiffEq.jl#1468
• More compile time fixes OrdinaryDiffEq.jl#1469
• remove diffcache DiffEqBase.jl#688
• reduce compile times by specializing broadcasts to loops DiffEqBase.jl#696
• remove reduce from array dispatch to reduce compile time DiffEqBase.jl#697
• attempt to precompile linsolve DiffEqBase.jl#698
• specialize ODEProblem remake for compile times SciMLBase.jl#95
• use ForwardDiff chunk heuristic JuliaDiff/SparseDiffTools.jl#147
• Make inference's job much easier by avoiding map JuliaDiff/SparseDiffTools.jl#149
• Force RecursiveFactorization to cache a bunch of precompiles JuliaLinearAlgebra/RecursiveFactorization.jl#29
• safer precompile JuliaLinearAlgebra/RecursiveFactorization.jl#30
• Try and force precompilation JuliaSIMD/TriangularSolve.jl#8
• Define prepare_alg dispatch for OrdinaryDiffEq AD algorithms OrdinaryDiffEq.jl#1470 (about to merge after a few more fixes)

with some bonus PRs and issues like:

• Remove single argument Pair constructor which invalidates sbromberger/LightGraphs.jl#1581
• remove rule for identity JuliaDiff/DiffRules.jl#64
• 0-element Vector{Base.Stacktraces.StackFrame} JuliaDebug/Cthulhu.jl#184
• Handle empty backtraces JuliaDebug/Cthulhu.jl#185
• Truncated stacktraces on Windows for specific code JuliaLang/julia#41750
• codegen: fix emission criteria for some special functions JuliaLang/julia#41813

Show me some results

The net result is something like this. On non-stiff ODEs, compile times dropped from about 5 seconds to sub 1 second, and on stiff ODEs compile times dropped from about 22 seconds to 2.5 seconds. The tests are things like:

SciML/OrdinaryDiffEq.jl#1465

using OrdinaryDiffEq, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Tsit5()
tinf = @snoopi_deep solve(prob,alg)
itrigs = inference_triggers(tinf)
itrig = itrigs[13]
ascend(itrig)
@time solve(prob,alg)

v5.60.2
InferenceTimingNode: 1.249748/4.881587 on Core.Compiler.Timings.ROOT() with 2 direct children

Before
InferenceTimingNode: 1.136504/3.852949 on Core.Compiler.Timings.ROOT() with 2 direct children

Without `@turbo`
InferenceTimingNode: 0.956948/3.460591 on Core.Compiler.Timings.ROOT() with 2 direct children

With `@inbounds @simd`
InferenceTimingNode: 0.941427/3.439566 on Core.Compiler.Timings.ROOT() with 2 direct children

With `@turbo`
InferenceTimingNode: 1.174613/11.118534 on Core.Compiler.Timings.ROOT() with 2 direct children

With `@inbounds @simd` everywhere
InferenceTimingNode: 0.760500/1.151602 on Core.Compiler.Timings.ROOT() with 2 direct children

# Today, a week after that PR
InferenceTimingNode: 0.634172/0.875295 on Core.Compiler.Timings.ROOT() with 1 direct children 🎉 
(it automatically does the emoji because the computer is happy too)

You read this as, it used to take 1.25 seconds for inference and 4.88 seconds for compilation in full, but now it's 0.63 and 0.88.

and SciML/DiffEqBase.jl#698

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)

using OrdinaryDiffEq, SnoopCompile
prob = ODEProblem(lorenz,u0,tspan)

alg = Rodas5()
tinf = @snoopi_deep solve(prob,alg)

After basic precompilation (most of the above PRs):
InferenceTimingNode: 1.460777/16.030597 on Core.Compiler.Timings.ROOT() with 46 direct children

After fixing precompilation of the LU-factorization tools:
InferenceTimingNode: 1.077774/2.868269 on Core.Compiler.Timings.ROOT() with 11 direct children

So that's the good news. The bad news.

The precompilation results do not always generalize

Take for example #785:

using DifferentialEquations, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Rodas5()
tinf = @snoopi_deep solve(prob,alg)

InferenceTimingNode: 1.535779/13.754596 on Core.Compiler.Timings.ROOT() with 7 direct children

"But Chris, I thought you just said that was sub 3 seconds, not 13.75 seconds compile time!". Well, that's with using OrdinaryDiffEq instead of using DifferentialEquations. And we see this when DiffEqSensitivity.jl or DiffEqFlux.jl gets involved.

So compile times are "a lot better"*, and the "+* are right now necessary when saying that. We need to fix that aspect of it.

How can I as a user help?

Good question, thanks for asking! Sharing profiles is extremely helpful. Take another look at SciML/DiffEqBase.jl#698 (comment) . What was ran was:

using OrdinaryDiffEq, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Rodas5()
tinf = @snoopi_deep solve(prob,alg)

using ProfileView
ProfileView.view(flamegraph(tinf))

What this was saying was that the vast majority of the compile time was because the DEFAULT_LINSOLVE calling RecursiveFactorization.jl was not precompiling. Since we use our own full Julia-based BLAS/LAPACK stack, that gave a full 13 seconds of compilation since it would compile RecursiveFactorization.jl, TriangularSolve.jl, etc. in sequence on each first solve call of a session. This allowed us to identify the issue can create a tizzy of PRs that finally made that get cached.

If you check the DifferentialEquations.jl compile times, you'll see that part is back. Why won't it compile? Well that's a harder question, discussed in SciML/DiffEqBase.jl#698 (comment), with the fix in SciML/DiffEqBase.jl#698 (comment), but apparently gets invalidated. If that all doesn't make sense to you, that's fine! But if you can help us narrow in on what the real issues are, that will help us immensely.

And of course... you can always give us a sponsor and star the repos to help 😅 . But seriously though, I hope we can start using SciML funds to start scouring our repos and get a lot of people fixing compile time issues. More on that soon... very soon... 🤐

What are some helpful tricks and knowledge to share?

Yeah, what were our tricks? Well the big one is forcing compilation in using calls through small solves. In many cases compilation is solved by just putting a prototype solve inside of the package itself. For example, take a look at the precompile section of OrdinaryDiffEq.jl (https://github.com/SciML/OrdinaryDiffEq.jl/blob/v5.61.1/src/OrdinaryDiffEq.jl#L175-L193)

let
    while true
      function lorenz(du,u,p,t)
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
      end
      lorenzprob = ODEProblem(lorenz,[1.0;0.0;0.0],(0.0,1.0))
      solve(lorenzprob,Tsit5())
      solve(lorenzprob,Vern7())
      solve(lorenzprob,Vern9())
      solve(lorenzprob,Rosenbrock23())(5.0)
      solve(lorenzprob,TRBDF2())
      solve(lorenzprob,Rodas4(autodiff=false))
      solve(lorenzprob,KenCarp4(autodiff=false))
      solve(lorenzprob,Rodas5())
      break
    end
  end

Lorenz equation takes nanoseconds to solve, so we take this equation and solve it a few times at using time, which will then trigger precompilation of as many functions hit in that call stack as Julia will allow for. For some things, the reason why they are not precompiled is simply because they are never called during using time, so a quick fix for many of the compile time issues is to simply add a short little statement like this to using time and then Julia will cache its results. For everything else, there's Mastercard, and it'll be much more costly to solve, so those will need issues and the experts, and possibly some Base compiler changes. But we should at least grab all of the low hanging fruit ASAP.

This is actually a necessary condition for getting precompilation, since if a method is never called in using then it will never precompile, so this is a first step among many.

A major part of the solution was avoiding codegen when unnecessary. If you take a look at SciML/OrdinaryDiffEq.jl#1465, you'll see things like:

@muladd function perform_step!(integrator, cache::Tsit5Cache{<:Array}, repeat_step=false)
  @unpack t,dt,uprev,u,f,p = integrator
  uidx = eachindex(integrator.uprev)
  @unpack c1,c2,c3,c4,c5,c6,a21,a31,a32,a41,a42,a43,a51,a52,a53,a54,a61,a62,a63,a64,a65,a71,a72,a73,a74,a75,a76,btilde1,btilde2,btilde3,btilde4,btilde5,btilde6,btilde7 = cache.tab
  @unpack k1,k2,k3,k4,k5,k6,k7,utilde,tmp,atmp = cache
  a = dt*a21
  @inbounds @simd ivdep for i in uidx
    tmp[i] = uprev[i]+a*k1[i]
  end
  f(k2, tmp, p, t+c1*dt)
  @inbounds @simd ivdep for i in uidx
    tmp[i] = uprev[i]+dt*(a31*k1[i]+a32*k2[i])
  end
...

i.e. new solver dispatches specifically for Array, to avoid dispatches like:

@muladd function perform_step!(integrator, cache::Tsit5Cache, repeat_step=false)
  @unpack t,dt,uprev,u,f,p = integrator
  @unpack c1,c2,c3,c4,c5,c6,a21,a31,a32,a41,a42,a43,a51,a52,a53,a54,a61,a62,a63,a64,a65,a71,a72,a73,a74,a75,a76,btilde1,btilde2,btilde3,btilde4,btilde5,btilde6,btilde7 = cache.tab
  @unpack k1,k2,k3,k4,k5,k6,k7,utilde,tmp,atmp = cache
  a = dt*a21
  @.. tmp = uprev+a*k1
  f(k2, tmp, p, t+c1*dt)
  @.. tmp = uprev+dt*(a31*k1+a32*k2)
...

The reason is that compile time profiling showcased that the major contributor was these https://github.com/YingboMa/FastBroadcast.jl code generation steps. Base Broadcast too is a major contributor to compile times. So at least on the pieces of code that 99% of users are using, we just expanded them out by hand, forced them to precompile, and that gave the 5 seconds to 1 second compile time change. I wouldn't say I am recommending you shouldn't use broadcast, but this is something useful to keep in mind. Broadcast is more generic, so you have to pay in compile time to use it. If you're willing to maintain the code, doing if u isa Array or making a separate dispatch can remove that factor.

Map is also a major contributor. We actually knew this was the case already, since there was a map call which changed Pumas compile times on a small example from 3 seconds to over 40 (SciML/SciMLBase.jl#45). Removing map calls factored into these changes as well (JuliaDiff/SparseDiffTools.jl#149). Again, this is something that could/should be handled at the Base compiler level, but we wanted as much of a fix ASAP so this was a nice and easy change which was within our power to ship in a week, and you can do the same.

The last thing, and the major step forward, was SciML/DiffEqBase.jl#698 (comment) . As it explains, using a function barrier can cause inference to not know what functions it will need in a call, which makes Julia seem to compile a whole lot of junk. That can probably get fixed at the Base compiler level to some extent, as the example there was spending 13 seconds compiling (::DefaultLinSolve)(::Vector{Float64}, ::Any, ::Vector{Float64}, ::Bool) junk which nobody could ever call, since any call to that function would specialize on the second argument (a matrix type) and so it should've been compiling (::DefaultLinSolve)(::Vector{Float64}, ::Matrix{Float64}, ::Vector{Float64}, ::Bool). But since that call was already precompiled, if inference ends up good enough that it realized it would call that instead, then bingo compile times dropped from 16 seconds to sub 3. That's a nice showcase that the easiest way to fix compile time issues may be to fix inference issues in a package, because that can make Julia narrow down what methods it needs to compile more, and then it may have a better chance of hitting the pieces that you told it to precompile (via a using-time example in a precompile.jl).

However, even if you have dynamism, you can still improve inference. The DiffEq problem was that ForwardDiff has a dynamically chosen chunksize based on the size of the input ODEs u. We previously would defer this calculation of the chunksize until the caches for the Jacobian were built, so then all dual number caches would have a where N on the chunk size from inference. But a function barrier makes the total runtime cost 100ns, so whatever that's solved right? But because all of the caches have this non-inference of the chunksize, Julia's compiler could not determine all of the N's were the same, so inference would see far too many types in expressions and just give up. That's how A::Matrix{Float64} became Any in the inference passes. When Julia hits the function barrier, it will cause inference to trigger again, in which case inside the function it will now be type-stable, so again no runtime cost, but at the first compile it will compile all possible methods it may ever need to call... which is a much bigger universe than what we actually hit. We noticed this was the case because if the user manually put in a chunksize, i.e. Rodas5(chunk_size = 3), then this all went away. So what we did was setup a hook so that the moment solve is called, it goes "have you defined the chunk size? If not, let's figure it out right away and then hit __solve. By doing this function barrier earlier, it at least knows that the chunk size should always match the N type parameter of the solver algorithm, so all of the Ns are the same, which makes it realize there's less types and use more of the compile caches before.

That one is harder than the others, but it makes the more general point that caching compilation is only good if inference is good enough.

And of course, check out Tim Holy's workshop and JuliaCon video which is immensely helpful at getting started.

https://www.youtube.com/watch?v=rVBgrWYKLHY
https://www.youtube.com/watch?v=wXRMwJdEjX4

What are the next steps?

Well, the next steps are to get the compile times below 0.1 seconds everywhere of course. I've already identified some issues to solve:

• DifferentialEquations.jl invalidates OrdinaryDiffEq.jl precompile caches for lu-factorization #785
• Get more things precompiling DiffEqFlux.jl#604
• start trying to precompile and reduce compile times SciMLSensitivity.jl#471
• How can we take the non-stiff ODE solvers even lower? There's pieces in the profile like Base.Logging that could probably get added to the Julia system image, etc. This needs a full investigation by someone who really knows their stuff.
• Can we get effective CI for finding compile time regressions?

Are there more things just missing some simple precompiles? Are there some major threads we're missing? Help us identify where all of this is so we can solve it in full.

Compile times, here we come!

[ChrisRackauckas]

Posted to Discourse as well: https://discourse.julialang.org/t/22-seconds-to-3-and-now-more-lets-fix-all-of-the-differentialequations-jl-universe-compile-times/66313

[ChrisRackauckas]

• Fix ~50 invalidations stemming from modules_to_be_loaded JuliaLang/julia#41878
• Fix ~370 invalidations from Expr(:ncat, ...) pretty-printing JuliaLang/julia#41877
• Fix ~160 invalidations from partially-inferred kwargs to Header JuliaIO/Tar.jl#123
• Fix a handful of invalidations in expression-checking JuliaLang/Downloads.jl#138
• Fix invalidations from Pair(obj) methods JuliaLang/Pkg.jl#2695
• Delete convert(::Type{Any}, ::UpperBoundedInteger) JuliaSIMD/LoopVectorization.jl#316

@timholy I think gathering them all here will make it easier to package up in the end.

[timholy]

• Improve inferrability of isor FluxML/MacroTools.jl#166
• Eliminate fallback reduce_empty JuliaLang/julia#41885
• Fix a couple of invalidations from extending == JuliaLang/Pkg.jl#2699
• Narrow convert methods to reduce invalidations JuliaAlgebra/MultivariatePolynomials.jl#172
• Use === in more places to reduce invalidation FluxML/MacroTools.jl#167
• Small improvement in inferrability JuliaPlots/RecipesBase.jl#82
• Small tweaks to prevent == invalidation JuliaArrays/StaticArrays.jl#950
• Various inferrability improvements JuliaLang/julia#41917
• Invalidations JuliaSIMD/CPUSummary.jl#3 (originally the CPUSummary issue identified in Attribution for "insert_backedges" invalidations JuliaLang/julia#41913)

[ChrisRackauckas]

Would moving dependencies over to say ExprTools.jl be helpful? I've known since SciML/MuladdMacro.jl#9 that MacroTools.jl can have some pretty major performance issues, so I've been curious if it's worth the engineering effort to move away from it at a larger scale. @oxinabox ?

[timholy]

Would moving dependencies over to say ExprTools.jl be helpful?

I don't really know, I'm not a big user of such packages (I tend to "roll my own" when I need stuff like that).

I think I pretty much have the invalidations fixed wrt to OrdinaryDiffEq (i.e., the ones you didn't take care of on your own), so I am starting to look at DifferentialEquations. The good news is that there appear to be only two substantive problems:

• Unitful: this promote_typejoin method triggers ~1500 invalidations (mostly via CSTParser). That's a bit of a dicey method but I don't think they're breaking any rules. It would be nice to come up with an alternative, though, because promote_typejoin is very close to the typesystem and it's quite nice to leave it a singleton method if possible. It only enables one test to pass, and the test fails if you change it slightly, so I think we could make a case for removing it (especially if we can give them something that works better). This method also bothers me because it doesn't actually construct a Quantity. Despite these issues, Unitful is a sophisticated, high-quality, well-developed package but it pushes the boundaries of the type system (it kinda has to). Nevertheless we're seeing some consequences from a few of its edgier choices.
• MultivariatePolynomials: these are trouble. The first one is the same near-piracy that we eliminated from LV. The second looks harder. The package has a lot of ambiguities so fixing things is a bit challenging. UPDATE: I may have fixed it, see Narrow convert methods to reduce invalidations JuliaAlgebra/MultivariatePolynomials.jl#172.

[oxinabox]

Would moving dependencies over to say ExprTools.jl be helpful?

ExprTools should be preferred for things that it can do.
Because it is more correct.
(There are weird edge-cases MacroTools gets wrong)
But the only thing it can do is splitdef/combinedef.
It also has no dependencies.

For prewalk and post-walk those are super easy to define.
I think the code that uses prewalk/postwalk is often clearer if it is written directly recursively than if it uses them anyway.

For pattern matching on Expr, maybe use MLStyle.jl?
I have heard it is much faster.

[timholy]

Tada!

julia> using DifferentialEquations, SnoopCompile

julia> function lorenz(du,u,p,t)
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
       end
lorenz (generic function with 1 method)

julia> u0 = [1.0;0.0;0.0]
3-element Vector{Float64}:
 1.0
 0.0
 0.0

julia> tspan = (0.0,100.0)
(0.0, 100.0)

julia> prob = ODEProblem(lorenz,u0,tspan)
ODEProblem with uType Vector{Float64} and tType Float64. In-place: true
timespan: (0.0, 100.0)
u0: 3-element Vector{Float64}:
 1.0
 0.0
 0.0

julia> alg = Tsit5()
Tsit5()

julia> tinf = @snoopi_deep solve(prob,alg)
InferenceTimingNode: 1.061332/1.436110 on Core.Compiler.Timings.ROOT() with 1 direct children

It involves a lot of custom stuff:

(@v1.8) pkg> st
Status `~/.julia/environments/v1.8/Project.toml`
 [1520ce14] AbstractTrees v0.3.4
 [6e4b80f9] BenchmarkTools v1.1.3
 [2a0fbf3d] CPUSummary v0.1.2 `~/.julia/dev/CPUSummary`
 [00ebfdb7] CSTParser v3.2.4
 [d360d2e6] ChainRulesCore v1.3.0 `~/.julia/dev/ChainRulesCore`
 [da1fd8a2] CodeTracking v1.0.6
 [3da002f7] ColorTypes v0.10.12
 [5ae59095] Colors v0.12.8
 [a2441757] Coverage v1.3.2
 [f68482b8] Cthulhu v2.2.0 `~/.julia/dev/Cthulhu`
 [31a5f54b] Debugger v0.6.8
 [2b5f629d] DiffEqBase v6.72.0 `~/.julia/dev/DiffEqBase`
 [b552c78f] DiffRules v1.3.0 `~/.julia/dev/DiffRules`
 [0c46a032] DifferentialEquations v6.19.0 `~/.julia/dev/DifferentialEquations`
 [e30172f5] Documenter v0.27.5
 [7876af07] Example v0.5.3
 [f6369f11] ForwardDiff v0.10.19
 [86fae568] ImageView v0.10.13
 [a98d9a8b] Interpolations v0.13.4
 [92d709cd] IrrationalConstants v0.1.0
 [c3a54625] JET v0.4.6
 [033835bb] JLD2 v0.4.13
 [aa1ae85d] JuliaInterpreter v0.8.20
 [89398ba2] LocalRegistry v0.3.2
 [bdcacae8] LoopVectorization v0.12.64 `~/.julia/dev/LoopVectorization`
 [6f1432cf] LoweredCodeUtils v2.1.2 `~/.julia/dev/LoweredCodeUtils`
 [1914dd2f] MacroTools v0.5.7 `~/.julia/dev/MacroTools`
 [dbb5928d] MappedArrays v0.4.1
 [85b6ec6f] MethodAnalysis v0.4.4
 [102ac46a] MultivariatePolynomials v0.3.18 `~/.julia/dev/MultivariatePolynomials`
 [d8a4904e] MutableArithmetics v0.2.20
 [6fe1bfb0] OffsetArrays v1.10.5
 [1dea7af3] OrdinaryDiffEq v5.61.1 `~/.julia/dev/OrdinaryDiffEq`
 [14b8a8f1] PkgTemplates v0.7.17
 [05854315] PrecompileCacheUtils v0.1.0 `../../dev/PrecompileCacheUtils`
 [c46f51b8] ProfileView v0.6.10
 [92933f4c] ProgressMeter v1.7.1
 [d330b81b] PyPlot v2.9.0
 [3cdcf5f2] RecipesBase v1.1.1 `~/.julia/dev/RecipesBase`
 [f2c3362d] RecursiveFactorization v0.2.2 `~/.julia/dev/RecursiveFactorization`
 [0f4576a4] RegistryCompatTools v0.1.0 `~/.julia/dev/RegistryCompatTools`
 [295af30f] Revise v3.1.19 `~/.julia/dev/Revise`
 [0bca4576] SciMLBase v1.18.4 `~/.julia/dev/SciMLBase`
 [aa65fe97] SnoopCompile v2.7.1 `~/.julia/dev/SnoopCompile`
 [e2b509da] SnoopCompileCore v2.7.1 `~/.julia/dev/SnoopCompile/SnoopCompileCore`
 [47a9eef4] SparseDiffTools v1.16.4 `~/.julia/dev/SparseDiffTools`
 [aedffcd0] Static v0.3.0 `~/.julia/dev/Static`
 [90137ffa] StaticArrays v1.2.12 `~/.julia/dev/StaticArrays`
 [4c63d2b9] StatsFuns v0.9.9
 [d5829a12] TriangularSolve v0.1.3 `~/.julia/dev/TriangularSolve`
 [1986cc42] Unitful v1.9.0 `~/.julia/dev/Unitful`

This is a bespoke build of Julia 1.8 (with several of my pending PRs) and tweaks to quite a few packages, including deleting the problematic Unitful method mentioned in #786 (comment). I'll see about trying to get the rest of this out there and merged to master,

Interestingly, this does not solve the CPUSummary issue, and yet I'm still getting great compile times. So we can afford to keep looking at this issue. CC @chriselrod.

[ChrisRackauckas]

well, well, 👏 👏 I'm impressed haha.

[timholy]

I can get this using the released versions of packages too, with the exceptions listed in #786 (comment) and the change to Unitful. I think the really big ones are (1) the reduce_empty fallback deletion, (2) the MultivariatePolynomials convert narrowing, and (3) the not-yet pushed Unitful change. (EDIT: hah, even the Unitful change isn't essential.) Plus much of the merged stuff to Julia 1.8.

We'll want to test 1.7 soon and see if you're still able to get good compile times, and if not possibly backport the essential PRs.

[ChrisRackauckas]

SciML/ModelingToolkit.jl#1215 highlights a parallel issue that once the symbolic tools get involved, there's a whole extra can of worms to consider. But that might have to wait until a later day.

[ChrisRackauckas]

timholy/SnoopCompile.jl#262

[ma-sadeghi]

Does inference/compilation times reported here include the time it takes to run using XXX?

[ChrisRackauckas]

No, not the using time, but the using time is relatively unchanged.

[ChrisRackauckas]

With:

SciML/OrdinaryDiffEq.jl#1535
SciML/OrdinaryDiffEq.jl#1521
JuliaDiff/SparseDiffTools.jl#168
JuliaDiff/SparseDiffTools.jl#167

Passing a chunk size is now down to 2 inference triggers total on static array code, both of which are a bit confusing.

using DifferentialEquations, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Rodas5(chunk_size = Val{3}())
tinf = @snoopi_deep solve(prob,alg)
solve(prob,alg)

julia> itrigs = inference_triggers(tinf)
2-element Vector{InferenceTrigger}:
 Inference triggered to call setindex!(::Vector{Tuple{Float64, Float64, Float64}}, ::Tuple{Bool, Bool, Bool}, ::Int64) from generate_chunked_partials (C:\Users\accou\.julia\dev\SparseDiffTools\src\differentiation\compute_jacobian_ad.jl:75) with specialization SparseDiffTools.generate_chunked_partials(::Vector{Float64}, ::UnitRange{Int64}, ::Val{3})
 Inference triggered to call OrdinaryDiffEq.jacobian2W!(::Matrix{Float64}, ::LinearAlgebra.UniformScaling{Bool}, ::Float64, ::Matrix{Float64}, ::Bool) called from toplevel

That's from what used to be 46 inference spots, now there should just be 0 or 1 depending on if the chunk size is passed.

Could constant prop fix chunk size inference?

[ChrisRackauckas]

JuliaLang/julia#43370 is a Julia Base PR that fixes #785 and halves the FTTP

[ChrisRackauckas]

#785 was solved by removing an invalidation in Symbolics: JuliaSymbolics/Symbolics.jl#471 . 🎉 Now using DifferentialEquations (or anything with Symbolics) doesn't invalidate the compile time fixes.

[ChrisRackauckas]

ForwardDiff compile time test problem:

using OrdinaryDiffEq, SnoopCompile, ForwardDiff

function lotka_volterra!(du, u, p, t)
  x, y = u
  α, β, δ, γ = p
  du[1] = dx = α*x - β*x*y
  du[2] = dy = -δ*y + γ*x*y
end

# Initial condition
u0 = [1.0, 1.0]

# Simulation interval and intermediary points
tspan = (0.0, 10.0)
tsteps = 0.0:0.1:10.0

# LV equation parameter. p = [α, β, δ, γ]
p = [1.5, 1.0, 3.0, 1.0]

# Setup the ODE problem, then solve
prob = ODEProblem(lotka_volterra!, u0, tspan, p)
sol = solve(prob, Tsit5())

function loss(p)
  sol = solve(prob, Tsit5(), p=p, saveat = tsteps)
  sum(abs2, sol.-1)
end

tinf = @snoopi_deep ForwardDiff.gradient(loss,p)

using ProfileView
ProfileView.view(flamegraph(tinf))

A lot of this is from ForwardDiff not caching some compiles. @chriselrod let's take a look at that.

[chriselrod]

A lot of this is from ForwardDiff not caching some compiles. @chriselrod let's take a look at that.

As discussed on slack, tagging is likely part of the problem.

A solution I'd suggest is defining a method for some entry point functions like solve to convert from the Dual input type to custom tags defined inside DiffEq.
The trick of course would be to avoid allocating and avoid reinterpret for Array inputs, which would still carry the old type.
My suggestion there would be to GC.@preserve, cast the pointer, and then use an array type wrapping the pointer.

Basically, add a solve method for Dual{T,V,N}s that automatically casts to Dual{DiffEqTagX,V,N} (and of course extend this to nested duals).
The trickiest part of this is probably handling the return values correctly. Probably best to allocate them in the correct return type, then GC.@preserve and cast for all the internal code filling it.

Perhaps it is best to only support StaticArrays initially, so that you don't have to worry about these conversions.

[ChrisRackauckas]

Basically, add a solve method for Dual{T,V,N}s that automatically casts to Dual{DiffEqTagX,V,N} (and of course extend this to nested duals).

That would go in the high level solve.jl in DiffEqBase.jl

For the internal duals, let's make an option in algorithms to use standardtag=true by default, and then just use DiffEqTag internally.

[ChrisRackauckas]

Forward mode AD is a lot faster now:

SciML/DiffEqBase.jl#722
SciML/OrdinaryDiffEq.jl#1546
SciML/OrdinaryDiffEq.jl#1544
SciML/OrdinaryDiffEq.jl#1540

using OrdinaryDiffEq, SnoopCompile, ForwardDiff

lorenz = (du,u,p,t) -> begin
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]; tspan = (0.0,100.0);
prob = ODEProblem(lorenz,u0,tspan); alg = Rodas5();
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)

Before:

#First
InferenceTimingNode: 1.849625/14.538148 on Core.Compiler.Timings.ROOT() with 32 direct children

#Second
InferenceTimingNode: 1.531660/4.170409 on Core.Compiler.Timings.ROOT() with 12 direct children

After:

#First
InferenceTimingNode: 1.181086/3.320321 on Core.Compiler.Timings.ROOT() with 32 direct children

#Second
InferenceTimingNode: 0.998814/1.650488 on Core.Compiler.Timings.ROOT() with 11 direct children

[ChrisRackauckas]

@timholy is going to beat all of us in this compile time race by attempting JuliaLang/julia#42016

[ChrisRackauckas]

Chunked out some invalidations: SciML/DiffEqBase.jl#723 and the CLIMA folks are happy CliMA/TurbulenceConvection.jl#673.

@timholy is there an easy way to figure out which invalidations you should prioritize?

[ChrisRackauckas]

I took some hints from @charleskawczynski and went invalidation hunting:

# From: https://timholy.github.io/SnoopCompile.jl/stable/snoopr/
using SnoopCompile
invalidations = @snoopr begin
    using DifferentialEquations

    function lorenz(du,u,p,t)
     du[1] = 10.0(u[2]-u[1])
     du[2] = u[1]*(28.0-u[3]) - u[2]
     du[3] = u[1]*u[2] - (8/3)*u[3]
    end
    u0 = [1.0;0.0;0.0]
    tspan = (0.0,100.0)
    prob = ODEProblem(lorenz,u0,tspan)
    alg = Rodas5()
    tinf = solve(prob,alg)
end;

trees = SnoopCompile.invalidation_trees(invalidations);

@show length(SnoopCompile.uinvalidated(invalidations)) # show total invalidations

show(trees[end]) # show the most invalidated method

# Count number of children (number of invalidations per invalidated method)
n_invalidations = map(trees) do methinvs
    SnoopCompile.countchildren(methinvs)
end

import Plots
Plots.plot(
    1:length(trees),
    n_invalidations;
    markershape = :circle,
    xlabel = "i-th method invalidation",
    label = "Number of children per method invalidations",
)

Resulting in downstream PRs to remove all of the largest offenders:

• Remove major invalidations from == JuliaDiff/ChainRulesCore.jl#524 (see Heavy invalidations from == overloads JuliaDiff/ChainRulesCore.jl#523)
• Fix some major invalidations to improve compile times PainterQubits/Unitful.jl#509

Except for JuliaFolds/InitialValues.jl#63, the last of which I'll want @tkf's help figuring out.

The PRs handle the top 3 at least, so that entire spike should be deleted.

Edit: JuliaFolds/InitialValues.jl#64 joined the fray

[ChrisRackauckas]

This is the current compile time benchmarks:

using DifferentialEquations, SnoopCompile

function lorenz(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz,u0,tspan)
alg = Tsit5()
tinf = @snoopi_deep solve(prob,alg)
#InferenceTimingNode: 0.753838/1.076605 on Core.Compiler.Timings.ROOT() with 1 direct children

# New Session
alg = Rodas5(chunk_size = Val{3}(),linsolve = DifferentialEquations.OrdinaryDiffEq.RFLUFactorization())
#InferenceTimingNode: 1.069709/1.983534 on Core.Compiler.Timings.ROOT() with 7 direct children

# New Session
using OrdinaryDiffEq, SnoopCompile, ForwardDiff

lorenz = (du,u,p,t) -> begin
        du[1] = 10.0(u[2]-u[1])
        du[2] = u[1]*(28.0-u[3]) - u[2]
        du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]; tspan = (0.0,100.0);
prob = ODEProblem(lorenz,u0,tspan); alg = Rodas5();
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)
tinf = @snoopi_deep ForwardDiff.gradient(u0 -> sum(solve(ODEProblem(lorenz,u0,tspan),alg)), u0)

# First: InferenceTimingNode: 1.233956/3.789888 on Core.Compiler.Timings.ROOT() with 35 direct children
# Second: InferenceTimingNode: 1.023062/1.785363 on Core.Compiler.Timings.ROOT() with 15 direct children

[ChrisRackauckas]

Now looking at using times, #835 chunked that down. I then started hunting downstream libraries with big Requires:

• Remove ForwardDiff portion JuliaSIMD/Polyester.jl#55
• Remove Requires.jl usage JuliaSIMD/LoopVectorization.jl#372

While trying to track down using times I noticed I was missing some tooling. @timholy do you know of a way to make the profiler tell you which package is using Requires.jl time? https://discourse.julialang.org/t/profiling-requires-jl/73570 The profiler says that most of the remaining time is in Requires.jl, but I can't figure out where.

So I commented out the Requires.jl usage in Polyester, LoopVectorization, ArrayInterface, and DiffEqBase. Adding DiffEqBase and ArrayInterface to the list had zero change from just Polyester and LoopVectorization, which means the SciML libraries are not the Requires.jl users that are the trouble, and surprisingly neither is ArrayInterface.jl. I then looked at with and without a custom system image:

using PackageCompiler
create_sysimage(["StaticArrays","ForwardDiff"],sysimage_path="sysimage2.so")

and saw:

# Before PR
julia> @time using DifferentialEquations
  8.694322 seconds (24.77 M allocations: 1.736 GiB, 7.27% gc time, 17.92% compilation time)

# After PR
julia> @time using DifferentialEquations
  5.761738 seconds (18.00 M allocations: 1.327 GiB, 4.54% gc time, 10.38% compilation time)

# After PR + sysimage
julia> @time using DifferentialEquations
  4.565475 seconds (14.80 M allocations: 1.108 GiB, 6.05% gc time, 8.73% compilation time)

That lead to this Discourse thread on what to do about ForwardDiff and StaticArrays.jl contributing about one second of this time. The mysterious other 4.5 seconds in Requires.jl time is still elusive though... I need a Requires profiler!

[ChrisRackauckas]

Package load times have improved dramatically by doing ArrayInterfaceCore, and now GPUArraysCore. ArrayInterface changes was about 1 second off everything. Now GPUArrays is another second off. Numbers:

Before:

julia> @time_imports using RecursiveArrayTools
18.3 ms ┌ MacroTools
35.6 ms ┌ ZygoteRules
2.3 ms ┌ Compat
318.7 ms ┌ FillArrays
1320.3 ms ┌ StaticArrays
80.1 ms ┌ Preferences
86.5 ms ┌ JLLWrappers
373.9 ms ┌ LLVMExtra_jll
17.1 ms ┌ CEnum
485.5 ms ┌ LLVM
3.2 ms ┌ Adapt
2731.4 ms ┌ GPUArrays
11.8 ms ┌ DocStringExtensions
2.1 ms ┌ IfElse
72.5 ms ┌ RecipesBase
100.5 ms ┌ Static
6.0 ms ┌ ArrayInterfaceCore
164.4 ms ┌ ArrayInterface
13.0 ms ┌ ArrayInterfaceStaticArrays
208.8 ms ┌ ChainRulesCore
5503.5 ms RecursiveArrayTools

julia> @time_imports using RecursiveArrayTools
28.2 ms ┌ MacroTools
53.8 ms ┌ ZygoteRules
3.2 ms ┌ Compat
503.4 ms ┌ FillArrays
1715.4 ms ┌ StaticArrays
69.1 ms ┌ Preferences
103.9 ms ┌ JLLWrappers
500.3 ms ┌ LLVMExtra_jll
7.0 ms ┌ CEnum
265.1 ms ┌ LLVM
3.1 ms ┌ Adapt
2544.7 ms ┌ GPUArrays
16.1 ms ┌ DocStringExtensions
3.1 ms ┌ IfElse
265.8 ms ┌ RecipesBase
175.6 ms ┌ Static
8.1 ms ┌ ArrayInterfaceCore
222.9 ms ┌ ArrayInterface
3.6 ms ┌ ArrayInterfaceStaticArrays
175.3 ms ┌ ChainRulesCore
6228.5 ms RecursiveArrayTools

After:

julia> @time_imports using RecursiveArrayTools
30.9 ms ┌ MacroTools
54.5 ms ┌ ZygoteRules
2.3 ms ┌ Compat
299.2 ms ┌ FillArrays
1240.2 ms ┌ StaticArrays
7.3 ms ┌ DocStringExtensions
1.5 ms ┌ IfElse
43.5 ms ┌ RecipesBase
149.6 ms ┌ Static
5.2 ms ┌ ArrayInterfaceCore
195.5 ms ┌ ArrayInterface
3.3 ms ┌ Adapt
5.7 ms ┌ ArrayInterfaceStaticArrays
185.8 ms ┌ ChainRulesCore
10.2 ms ┌ GPUArraysCore
2169.1 ms RecursiveArrayTools

julia> @time_imports using RecursiveArrayTools
23.3 ms ┌ MacroTools
44.2 ms ┌ ZygoteRules
2.4 ms ┌ Compat
305.4 ms ┌ FillArrays
1015.0 ms ┌ StaticArrays
12.3 ms ┌ DocStringExtensions
2.0 ms ┌ IfElse
108.6 ms ┌ RecipesBase
134.0 ms ┌ Static
5.8 ms ┌ ArrayInterfaceCore
190.5 ms ┌ ArrayInterface
3.4 ms ┌ Adapt
5.2 ms ┌ ArrayInterfaceStaticArrays
273.1 ms ┌ ChainRulesCore
13.3 ms ┌ GPUArraysCore
2111.9 ms RecursiveArrayTools

@time_imports using OrdinaryDiffEq

Before:

11.2 ms ┌ MacroTools
20.1 ms ┌ ZygoteRules
16.0 ms ┌ Preferences
17.7 ms ┌ JLLWrappers
68.4 ms ┌ Rmath_jll
134.5 ms ┌ Rmath
3.7 ms ┌ Compat
1.5 ms ┌ NaNMath
3.5 ms ┌ Calculus
52.7 ms ┌ ChainRulesCore
55.6 ms ┌ ChangesOfVariables
1.3 ms ┌ OpenLibm_jll
2.3 ms ┌ InverseFunctions
4.8 ms ┌ DocStringExtensions
5.4 ms ┌ IrrationalConstants
1.5 ms ┌ CompilerSupportLibraries_jll
2.0 ms ┌ LogExpFunctions
1.7 ms ┌ OpenSpecFun_jll
29.7 ms ┌ SpecialFunctions
13.7 ms ┌ DualNumbers
121.9 ms ┌ HypergeometricFunctions
1.0 ms ┌ Reexport
281.1 ms ┌ StatsFuns
2.0 ms ┌ CpuId
412.8 ms ┌ StaticArrays
1.1 ms ┌ SIMDTypes
1.2 ms ┌ IfElse
35.6 ms ┌ Static
3.5 ms ┌ ArrayInterfaceCore
56.2 ms ┌ ArrayInterface
1.7 ms ┌ Adapt
45.7 ms ┌ OffsetArrays
48.9 ms ┌ ArrayInterfaceOffsetArrays
3.6 ms ┌ ArrayInterfaceStaticArrays
3.0 ms ┌ ManualMemory
538.4 ms ┌ LayoutPointers
57.3 ms ┌ CPUSummary
1.0 ms ┌ BitTwiddlingConvenienceFunctions
67.9 ms ┌ HostCPUFeatures
1038.9 ms ┌ VectorizationBase
0.9 ms ┌ IteratorInterfaceExtensions
2.6 ms ┌ TableTraits
41.7 ms ┌ ThreadingUtilities
59.1 ms ┌ PolyesterWeave
3.6 ms ┌ DiffResults
13.8 ms ┌ FunctionWrappers
1.8 ms ┌ CommonSubexpressions
2.0 ms ┌ DiffRules
113.9 ms ┌ ForwardDiff
6.1 ms ┌ SLEEFPirates
155.3 ms ┌ SIMDDualNumbers
7.8 ms ┌ OrderedCollections
74.5 ms ┌ DataStructures
1.4 ms ┌ MuladdMacro
13.6 ms ┌ KLU
26.2 ms ┌ LabelledArrays
30.2 ms ┌ PreallocationTools
1.7 ms ┌ Requires
1.2 ms ┌ StatsAPI
6.9 ms ┌ Distances
0.9 ms ┌ UnPack
144.2 ms ┌ FillArrays
1.3 ms ┌ DataValueInterfaces
113.2 ms ┌ LLVMExtra_jll
6.0 ms ┌ CEnum
121.8 ms ┌ LLVM
4.3 ms ┌ GPUArraysCore
825.5 ms ┌ GPUArrays
28.6 ms ┌ RecipesBase
1039.7 ms ┌ RecursiveArrayTools
2.8 ms ┌ DataAPI
1.5 ms ┌ SortingAlgorithms
8.4 ms ┌ Missings
38.7 ms ┌ StatsBase
1.6 ms ┌ ArrayInterfaceGPUArrays
5.0 ms ┌ CloseOpenIntervals
27.6 ms ┌ StrideArraysCore
37.6 ms ┌ Polyester
15.7 ms ┌ Tables
1.2 ms ┌ CommonSolve
2.5 ms ┌ ConstructionBase
1.0 ms ┌ TreeViews
294.0 ms ┌ SciMLBase
74.2 ms ┌ Setfield
348.4 ms ┌ LoopVectorization
14.0 ms ┌ FiniteDiff
89.4 ms ┌ TriangularSolve
22.0 ms ┌ IterativeSolvers
179.2 ms ┌ RecursiveFactorization
1074.1 ms ┌ NonlinearSolve
1.4 ms ┌ libblastrampoline_jll
24.6 ms ┌ GenericSchur
1.7 ms ┌ Parameters
23.7 ms ┌ PDMats
14.2 ms ┌ Krylov
7.9 ms ┌ NLSolversBase
20.0 ms ┌ LineSearches
1.8 ms ┌ Inflate
3.1 ms ┌ DensityInterface
5.6 ms ┌ QuadGK
243.1 ms ┌ Distributions
55.2 ms ┌ KrylovKit
66.9 ms ┌ LinearSolve
1.5 ms ┌ FastClosures
2.6 ms ┌ FastBroadcast
6.2 ms ┌ NLsolve
3.0 ms ┌ SimpleTraits
7.0 ms ┌ ArnoldiMethod
55.6 ms ┌ Graphs
62.5 ms ┌ VertexSafeGraphs
82.7 ms ┌ SparseDiffTools
131.3 ms ┌ DiffEqBase
7.3 ms ┌ ExponentialUtilities
5809.3 ms OrdinaryDiffEq

After:

SciML/RecursiveArrayTools.jl#213
JuliaArrays/ArrayInterface.jl#320
SciML/DiffEqNoiseProcess.jl#106
SciML/LinearSolve.jl#146
SciML/ExponentialUtilities.jl#91

@time_imports using OrdinaryDiffEq

10.4 ms ┌ MacroTools
17.7 ms ┌ ZygoteRules
15.6 ms ┌ Preferences
17.3 ms ┌ JLLWrappers
163.2 ms ┌ Rmath_jll
252.5 ms ┌ Rmath
3.0 ms ┌ Compat
1.3 ms ┌ NaNMath
2.8 ms ┌ Calculus
49.3 ms ┌ ChainRulesCore
51.3 ms ┌ ChangesOfVariables
1.2 ms ┌ OpenLibm_jll
2.1 ms ┌ InverseFunctions
4.1 ms ┌ DocStringExtensions
4.6 ms ┌ IrrationalConstants
1.3 ms ┌ CompilerSupportLibraries_jll
1.8 ms ┌ LogExpFunctions
1.7 ms ┌ OpenSpecFun_jll
15.0 ms ┌ SpecialFunctions
11.5 ms ┌ DualNumbers
98.0 ms ┌ HypergeometricFunctions
0.9 ms ┌ Reexport
372.3 ms ┌ StatsFuns
1.6 ms ┌ CpuId
413.5 ms ┌ StaticArrays
1.2 ms ┌ SIMDTypes
0.9 ms ┌ IfElse
34.7 ms ┌ Static
2.8 ms ┌ ArrayInterfaceCore
52.9 ms ┌ ArrayInterface
1.7 ms ┌ Adapt
40.4 ms ┌ OffsetArrays
43.5 ms ┌ ArrayInterfaceOffsetArrays
3.1 ms ┌ ArrayInterfaceStaticArrays
3.3 ms ┌ ManualMemory
529.6 ms ┌ LayoutPointers
56.7 ms ┌ CPUSummary
1.0 ms ┌ BitTwiddlingConvenienceFunctions
95.7 ms ┌ HostCPUFeatures
1036.7 ms ┌ VectorizationBase
0.8 ms ┌ IteratorInterfaceExtensions
2.4 ms ┌ TableTraits
41.7 ms ┌ ThreadingUtilities
59.5 ms ┌ PolyesterWeave
3.7 ms ┌ DiffResults
12.7 ms ┌ FunctionWrappers
1.3 ms ┌ CommonSubexpressions
2.0 ms ┌ DiffRules
116.5 ms ┌ ForwardDiff
6.1 ms ┌ SLEEFPirates
157.1 ms ┌ SIMDDualNumbers
7.0 ms ┌ OrderedCollections
71.3 ms ┌ DataStructures
1.3 ms ┌ MuladdMacro
13.7 ms ┌ KLU
25.4 ms ┌ LabelledArrays
28.3 ms ┌ PreallocationTools
1.5 ms ┌ Requires
1.2 ms ┌ StatsAPI
6.5 ms ┌ Distances
1.1 ms ┌ UnPack
141.8 ms ┌ FillArrays
1.2 ms ┌ DataValueInterfaces
24.0 ms ┌ RecipesBase
3.2 ms ┌ GPUArraysCore
60.3 ms ┌ RecursiveArrayTools
2.6 ms ┌ DataAPI
1.7 ms ┌ SortingAlgorithms
8.5 ms ┌ Missings
36.1 ms ┌ StatsBase
1.7 ms ┌ ArrayInterfaceGPUArrays
6.1 ms ┌ CloseOpenIntervals
25.8 ms ┌ StrideArraysCore
36.7 ms ┌ Polyester
15.5 ms ┌ Tables
1.1 ms ┌ CommonSolve
1.8 ms ┌ ConstructionBase
1.0 ms ┌ TreeViews
277.6 ms ┌ SciMLBase
71.2 ms ┌ Setfield
312.5 ms ┌ LoopVectorization
14.2 ms ┌ FiniteDiff
85.5 ms ┌ TriangularSolve
20.6 ms ┌ IterativeSolvers
111.1 ms ┌ RecursiveFactorization
945.4 ms ┌ NonlinearSolve
1.2 ms ┌ libblastrampoline_jll
24.0 ms ┌ GenericSchur
23.2 ms ┌ PDMats
1.4 ms ┌ Parameters
13.4 ms ┌ Krylov
6.4 ms ┌ NLSolversBase
16.7 ms ┌ LineSearches
1.3 ms ┌ Inflate
2.2 ms ┌ DensityInterface
4.0 ms ┌ QuadGK
297.9 ms ┌ Distributions
53.6 ms ┌ KrylovKit
63.2 ms ┌ LinearSolve
1.3 ms ┌ FastClosures
2.3 ms ┌ FastBroadcast
4.9 ms ┌ NLsolve
2.5 ms ┌ SimpleTraits
5.4 ms ┌ ArnoldiMethod
48.2 ms ┌ Graphs
53.2 ms ┌ VertexSafeGraphs
71.7 ms ┌ SparseDiffTools
59.6 ms ┌ DiffEqBase
136.5 ms ┌ ExponentialUtilities
4685.2 ms OrdinaryDiffEq

[ChrisRackauckas]

Current invalidations report:

using SnoopCompile
invalidations = @snoopr begin
    using OrdinaryDiffEq

    function lorenz(du, u, p, t)
        du[1] = 10.0(u[2] - u[1])
        du[2] = u[1] * (28.0 - u[3]) - u[2]
        du[3] = u[1] * u[2] - (8 / 3) * u[3]
    end
    u0 = [1.0; 0.0; 0.0]
    tspan = (0.0, 100.0)
    prob = ODEProblem{true,false}(lorenz, u0, tspan)
    alg = Rodas5()
    tinf = solve(prob, alg)
end;

trees = SnoopCompile.invalidation_trees(invalidations);

@show length(SnoopCompile.uinvalidated(invalidations)) # show total invalidations

show(trees[end]) # show the most invalidated method

# Count number of children (number of invalidations per invalidated method)
n_invalidations = map(trees) do methinvs
    SnoopCompile.countchildren(methinvs)
end

import Plots
Plots.plot(
    1:length(trees),
    n_invalidations;
    markershape=:circle,
    xlabel="i-th method invalidation",
    label="Number of children per method invalidations"
)

julia> show(trees[end])
inserting convert(::Type{DAEFunction}, f) in SciMLBase at C:\Users\accou\.julia\packages\SciMLBase\3fOCs\src\scimlfunctions.jl:3227 invalidated:
   mt_backedges:  1: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for error(::String, ::Expr) (0 children)
                  2: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for FileIO.querysym_all(::IOStream) (0 children)
                  3: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for JuliaInterpreter.compute_ssa_mapping_delete_statements!(::Core.CodeInfo, ::Vector{Int64}) (0 children)
                  4: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  5: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for SnoopCompile._show(::IOContext{IOBuffer}, ::SnoopCompileCore.InferenceTiming) (0 children)
                  6: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.mapreduce_empty_iter(::Function, ::Function, ::Vector{Expr}, ::Base.HasEltype) (0 children)
                  7: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::typeof(identity), ::Function) (1 children)
                  8: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(min)) (2 children)
                  9: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(max)) (6 children)
                 10: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(Base.add_sum)) (6 children)
                 11: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(|)) (22 children)
                 12: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::typeof(&)) (24 children)
                 13: signature Tuple{typeof(convert), Type{F} where F<:Function, Function} triggered MethodInstance for Base.MappingRF(::Function, ::Function) (1704 children)
                 14: signature convert(::Type{T}, x::T) where T in Base at Base.jl:61 (formerly convert(::Type{T}, x::T) where T in Base at Base.jl:61) triggered MethodInstance for Base.print_to_string(::Symbol, ::String, ::Expr, ::String) (1 children)
                 15: signature convert(::Type{T}, x::T) where T in Base at Base.jl:61 (formerly convert(::Type{T}, x::T) where T in Base at Base.jl:61) triggered MethodInstance for KrylovKit._set_num_threads_warn(::Int64) (1 children)
   backedges: 1: superseding convert(::Type{T}, x::T) where T in Base at Base.jl:61 with MethodInstance for convert(::Type{T}, ::T) where T<:Function (13 children)
   1 mt_cache
false

julia> show(trees[end-1])
inserting !(::Static.False) in Static at C:\Users\accou\.julia\packages\Static\sVI3g\src\Static.jl:427 invalidated:
   mt_backedges:   1: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::AbstractFloat, ::AbstractFloat) (0 children)
                   2: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isbadzero(::typeof(min), ::AbstractFloat) (0 children)
                   3: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                   4: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                   5: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                   6: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                   7: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                   8: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                   9: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  10: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Integer, ::Int64) (0 children)
                  11: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, _A, Tuple{Symbol, Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Nothing, ::Any, ::Symbol, ::Nothing, ::Int64) (0 children)
                  12: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:maxlog,), Tuple{Int64}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  13: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Tuple{ErrorException, Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}}, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{Tuple{ErrorException, Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}}}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  14: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97")(::Int64) (0 children)
                  15: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97"{typeof(iszero)})(::Any) (0 children)
                  16: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{<:Base.Order.ReverseOrdering{Base.Order.ForwardOrdering}, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}) (0 children)
                  17: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{<:Base.Order.ForwardOrdering, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}) (0 children)
                  18: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Float64}} where _A<:Base.Order.Ordering) (0 children)
                  19: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Int64) (0 children)
                  20: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, Base.ExceptionStack, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{Base.ExceptionStack}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  21: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.var"#handle_message#2"(::Base.Pairs{Symbol, ErrorException, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{ErrorException}}}, ::typeof(Base.CoreLogging.handle_message), ::Base.CoreLogging.SimpleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  22: signature Tuple{typeof(!), Any} triggered MethodInstance for Artifacts.process_overrides(::Dict{String, Any}, ::Base.UUID) (0 children)
                  23: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                  24: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                  25: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Any, ::Symbol, ::Any, ::Any, ::Any) (0 children)
                  26: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::LazyString, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  27: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  28: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (0 children)
                  29: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  30: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, Base.ExceptionStack, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{Base.ExceptionStack}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  31: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, ErrorException, Tuple{Symbol}, NamedTuple{(:exception,), Tuple{ErrorException}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Module, ::Symbol, ::Symbol, ::String, ::Int64) (0 children)
                  32: signature Tuple{typeof(!), Any} triggered MethodInstance for Logging.var"#handle_message#3"(::Base.Pairs{Symbol, _A, Tuple{Symbol, Symbol}, NamedTuple{names, T}} where {_A, names, T<:Tuple{Vararg{Any, N}}}, ::typeof(Base.CoreLogging.handle_message), ::Logging.ConsoleLogger, ::Base.CoreLogging.LogLevel, ::String, ::Nothing, ::Any, ::Symbol, ::Nothing, ::Int64) (0 children)
                  33: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.var"#add_nested_key!#24"(::Any, ::typeof(REPL.LineEdit.add_nested_key!), ::Dict, ::Char, ::Any) (0 children)
                  34: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.getEntry(::Dict{Char, Any}, ::String) (0 children)
                  35: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.LazilyInitializedFields.lazy_struct(::Expr) (0 children)
                  36: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float64}) (0 children)
                  37: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.at_disable_library_threading(::Function) (0 children)
                  38: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.at_disable_library_threading(::LinearAlgebra.var"#251#252") (0 children)
                  39: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.mightalias(::Vector, ::Vector{String}) (0 children)
                  40: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) (0 children)  
                  41: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Unsigned, ::Int64) (0 children)
                  42: signature Tuple{typeof(!), Any} triggered MethodInstance for showerror(::IOContext{Base.TTY}, ::MethodError) (0 children)
                  43: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.Types.Manifest(::Dict{String, Any}, ::String) (0 children)
                  44: signature Tuple{typeof(!), Any} triggered MethodInstance for Tar.var"#read_tarball#47"(::Vector{UInt8}, ::Base.DevNull, ::typeof(Tar.read_tarball), ::Function, ::Function, ::Base.Process) (0 children)
                  45: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.Registry.var"#11#14"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd) (0 children)
                  46: signature Tuple{typeof(!), Any} triggered MethodInstance for Tar.var"#read_tarball#47"(::Vector{UInt8}, ::Base.DevNull, ::typeof(Tar.read_tarball), ::Tar.var"#26#28"{Vector{UInt8}, Bool, Bool, Base.Process, String}, ::Function, ::Base.Process) (0 children)
                  47: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Pkg.PlatformEngines.var"#26#28"{String}, ::Cmd) (0 children)
                  48: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.mightalias(::Vector, ::Vector{Pkg.Types.PackageSpec}) (0 children)
                  49: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Any) (0 children)
                  50: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Tuple{Float64, Float64}}} where _A<:Base.Order.Ordering) (0 children)
                  51: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::Any, ::Any) (0 children)
                  52: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::Base.UUID, ::Any) (0 children)
                  53: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.Types.Manifest(::Dict{String, Any}, ::Base.DevNull) (0 children)     
                  54: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.Types.Manifest(::Dict{String, Any}, ::IOBuffer) (0 children)
                  55: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Union{Nothing, Pkg.Types.UpgradeLevel, VersionNumber, String, Pkg.Versions.VersionSpec}, ::Union{Nothing, Pkg.Types.UpgradeLevel, VersionNumber, String, Pkg.Versions.VersionSpec}) (0 children)
                  56: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Union{Missing, Float32}}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  57: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Union{Missing, Float64}}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  58: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Missing}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  59: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Float32}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  60: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::AbstractVector{Float64}, ::Integer, ::Integer, ::Base.Sort.InsertionSortAlg, ::Base.Order.ForwardOrdering) (0 children)
                  61: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sort!#8"(::Base.Sort.Algorithm, ::typeof(isless), ::typeof(identity), ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sort!), ::AbstractVector) (0 children)
                  62: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sort!#8"(::Base.Sort.Algorithm, ::typeof(isless), ::typeof(identity), ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sort!), ::Vector) (0 children)
                  63: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.API.var"#write_condensed_toml#182")(::Pkg.API.var"#153#184"{Set{String}}, ::Dict{String, Dict{String, Dates.DateTime}}, ::String) (0 children)
                  64: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.API.var"#write_condensed_toml#182")(::Pkg.API.var"#156#187"{Set{String}}, ::Dict{String, Dict{String, Dates.DateTime}}, ::String) (0 children)
                  65: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97")(::String) (0 children)
                  66: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.API.var"#write_condensed_toml#182")(::Pkg.API.var"#159#190"{Dict{String, Dict{String, Set{String}}}, Set{String}, Set{String}}, ::Dict{String, Dict{String, Dates.DateTime}}, ::String) (0 children)
                  67: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#96#97")(::Base.PkgId) (0 children)
                  68: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Tar.var"#83#86"{Base.DevNull, Bool, _A, String} where _A, ::Cmd) (0 children)
                  69: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Tar.var"#83#86"{Base.DevNull, Bool, Tar.var"#1#2", String}, ::Cmd) (0 children)
                  70: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::UInt8) (0 children)
                  71: signature Tuple{typeof(!), Any} triggered MethodInstance for ==(::Dict{String, Any}, ::Dict{String, Any}) (0 children)
                  72: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.mightalias(::Vector, ::Vector{Any}) (0 children)
                  73: signature Tuple{typeof(!), Any} triggered MethodInstance for Base._show_nonempty(::IOContext{Base.TTY}, ::AbstractMatrix, ::String, ::Bool, ::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}) (0 children)
                  74: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Int64}} where _A<:Base.Order.Ordering) (0 children)
                  75: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Perm{_A, Vector{Base.StackTraces.StackFrame}} where _A<:Base.Order.Ordering) (0 children)
                  76: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.REPLMode._completions(::String, ::Bool, ::Int64, ::Int64) (0 children)
                  77: signature Tuple{typeof(!), Any} triggered MethodInstance for allunique(::AbstractRange) (0 children)
                  78: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Char) (0 children)
                  79: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::GlobalRef, ::Any) (0 children)
                  80: signature Tuple{typeof(!), Any} triggered MethodInstance for Test.do_test_throws(::Test.ExecutionResult, ::Any, ::Any) (0 children)   
                  81: signature Tuple{typeof(!), Any} triggered MethodInstance for Test.eval_test(::Expr, ::Expr, ::LineNumberNode, ::Bool) (0 children)    
                  82: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float16}) (0 children)
                  83: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float32}) (0 children)
                  84: signature Tuple{typeof(!), Any} triggered MethodInstance for VSCodeServer.on_pkg_load(::Base.PkgId) (0 children)
                  85: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess(::Vector, ::Base.Broadcast.Broadcasted{Nothing, Tuple{Base.OneTo{Int64}}, _A, Tuple{Vector{Any}}} where _A) (0 children)
                  86: signature Tuple{typeof(!), Any} triggered MethodInstance for VSCodeServer.JuliaInterpreter.compute_ssa_mapping_delete_statements!(::Core.CodeInfo, ::Vector{Int64}) (0 children)
                  87: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess_args(::Vector, ::Tuple{Vector{Core.LineInfoNode}}) (0 children)
                  88: signature Tuple{typeof(!), Any} triggered MethodInstance for FileIO.__init__() (0 children)
                  89: signature Tuple{typeof(!), Any} triggered MethodInstance for FlameGraphs.var"#flamegraph!#8"(::Bool, ::Vector{Tuple{Symbol, Symbol}}, ::Int64, ::typeof(FlameGraphs.flamegraph!), ::LeftChildRightSiblingTrees.Node{FlameGraphs.NodeData}, ::Profile.StackFrameTree{Base.StackTraces.StackFrame}) (0 children)
                  90: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess(::Vector, ::Base.Broadcast.Broadcasted{Nothing, Tuple{Base.OneTo{Int64}}, _A, Tuple{Vector{Any}}} where _A) (0 children)
                  91: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Broadcast.preprocess_args(::Vector, ::Tuple{Vector{Core.LineInfoNode}}) (0 children)
                  92: signature Tuple{typeof(!), Any} triggered MethodInstance for JuliaInterpreter.compute_ssa_mapping_delete_statements!(::Core.CodeInfo, ::Vector{Int64}) (0 children)
                  93: signature Tuple{typeof(!), Any} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  94: signature Tuple{typeof(!), Any} triggered MethodInstance for SnoopCompile.__init__() (0 children)
                  95: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sortperm#12"(::Base.Sort.QuickSortAlg, ::Function, ::Function, ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sortperm), ::Vector{Float64}) (0 children)
                  96: signature Tuple{typeof(!), Any} triggered MethodInstance for StaticArrays.var"#s25#139"(::Any, ::Any, ::Any, ::Any) (0 children)      
                  97: signature Tuple{typeof(!), Any} triggered MethodInstance for FiniteDiff.__init__() (0 children)
                  98: signature Tuple{typeof(!), Any} triggered MethodInstance for Artifacts.var"#artifact_meta#12"(::Base.BinaryPlatforms.Platform, ::typeof(Artifacts.artifact_meta), ::String, ::Dict{String, Any}, ::String) (0 children)
                  99: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Int64, ::ForwardDiff.Dual{Ty}) where Ty (0 children)
                 100: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.match(::Expr, ::Symbol, ::Dict{Any, Any}) (0 children)        
                 101: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Tuple{Any, Vararg{Any}}) (1 children)
                 102: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Tuple{}) (1 children)
                 103: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isdelimited(::IOContext{IOBuffer}, ::Pair) (1 children)
                 104: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::LineNumberNode, ::Module, ::Any, ::Any, ::Expr) (1 children)
                 105: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::LineNumberNode, ::Module, ::Expr, ::Any, ::Expr) (1 children)
                 106: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.var"#add_nested_key!#24"(::Any, ::typeof(REPL.LineEdit.add_nested_key!), ::Dict, ::String, ::Any) (1 children)
                 107: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.AbstractPlatform) (1 children)
                 108: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEditREPL(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (1 children)
                 109: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.REPLMode.var"#command_is_focused#53"{Bool, Int64})() (1 children) 
                 110: signature Tuple{typeof(!), Any} triggered MethodInstance for VSCodeServer.add_code_to_repl_history(::String) (1 children)
                 111: signature Tuple{typeof(!), Any} triggered MethodInstance for Artifacts.process_overrides(::Dict{String, Any}, ::Base.UUID) (1 children)
                 112: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Int64, ::Any) (2 children)
                 113: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Dict{Char, Any}, ::Any) (2 children)
                 114: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Function}, ::Any, ::Any) (2 children)
                 115: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Vector{Dict}}, ::Vector, ::Any) (2 children)    
                 116: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Vector{String}}, ::Vector{String}, ::Any) (2 children)
                 117: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Base.UUID}, ::Base.UUID, ::Any) (2 children)
                 118: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Pkg.Types.Compat}, ::Any, ::Any) (2 children)   
                 119: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL._trimdocs(::Markdown.MD, ::Bool) (2 children)
                 120: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.var"#open#734"(::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, ::typeof(open), ::Function, ::Cmd) (3 children)
                 121: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Vector{String}, ::Any) (3 children)
                 122: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Dict{String, String}, ::Any) (3 children)
                 123: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Nothing, ::Any) (3 children)
                 124: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Any) (4 children)  
                 125: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Symbol}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 126: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Core.SimpleVector}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 127: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isdelimited(::IOContext{IOBuffer}, ::Pair{Symbol, Any}) (4 children)
                 128: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Union{Bool, String}}, ::Any, ::Any) (4 children)
                 129: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Union{Nothing, String}}, ::Any, ::Any) (4 children)
                 130: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pkg.Versions.VersionRange}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 131: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Pkg.REPLMode.CommandSpec}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 132: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Any}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 133: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Vector{Pkg.Types.Stage1}}, ::Vector{Pkg.Types.Stage1}, ::Any) (4 children)
                 134: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, String}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 135: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Base.UUID}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 136: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Vector{Base.UUID}}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 137: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{VersionNumber}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 138: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Base.BinaryPlatforms.AbstractPlatform}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 139: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Set{Int64}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 140: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Union{Nothing, Pkg.Resolve.ResolveLogEntry}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 141: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Base.UUID}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 142: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Pair{String, Pkg.Types.Compat}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 143: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Tuple{Union{Nothing, Base.UUID}, Union{Nothing, Pkg.Types.PackageSpec}, Union{Nothing, Pkg.Types.PackageSpec}}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 144: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Tuple{Base.UUID, String, String, VersionNumber}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 145: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Dates.DateTime, ::Any) (4 children)
                 146: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Base.BinaryPlatforms.Platform}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 147: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{REPL.REPLCompletions.Completion}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (4 children)
                 148: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.store!(::Dict{Any, Any}, ::Symbol, ::Expr) (4 children)       
                 149: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Union{Int64, Symbol}}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (5 children)
                 150: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.Platform, ::Base.BinaryPlatforms.Platform) (5 children)
                 151: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{_A, Base.Order.ForwardOrdering} where _A) (5 children)
                 152: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::VersionNumber, ::Any) (5 children)
                 153: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Base.SHA1, ::Any) (5 children)
                 154: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Bool, ::Any) (5 children)
                 155: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Nothing}, ::Nothing, ::Any) (6 children)        
                 156: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.getEntry(::Dict{Char, Any}, ::Union{Char, String}) (6 children)
                 157: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Base.UUID}, ::Any, ::Any) (7 children)
                 158: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Int64, ::Any) (7 children)
                 159: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#38#40")(::Core.MethodMatch) (7 children)
                 160: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.store!(::Dict{Any, Any}, ::Symbol, ::Any) (7 children)        
                 161: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{Pkg.API.var"#3#5", Base.Order.ForwardOrdering}) (8 children)
                 162: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.REPLMode.Command(::Pkg.REPLMode.Statement) (9 children)
                 163: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Any}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{Base.var"#874#880", Base.Order.ForwardOrdering}) (10 children)
                 164: signature Tuple{typeof(!), Any} triggered MethodInstance for MacroTools.store!(::Dict{Any, Any}, ::Symbol, ::Vector{Any}) (11 children)
                 165: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{Int64}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (12 children)
                 166: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.By{Pkg.Types.var"#30#32", Base.Order.ForwardOrdering}) (12 children)
                 167: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Int64}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (12 children)
                 168: signature Tuple{typeof(!), Any} triggered MethodInstance for show(::IO, ::TypeVar) (13 children)
                 169: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Tuple{Float64, Float64}}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (15 children)
                 170: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::Any, ::Any, ::Any, ::Any, ::Expr) (16 children)    
                 171: signature Tuple{typeof(!), Any} triggered MethodInstance for SnoopCompile.var"#next_julia_frame#65"(::Bool, ::Bool, ::typeof(SnoopCompile.next_julia_frame), ::Any, ::Int64, ::Int64) (17 children)
                 172: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.Platform) (18 children)
                 173: signature Tuple{typeof(!), Any} triggered MethodInstance for isequal(::Vector{Any}, ::Vector{Any}) (21 children)
                 174: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Union{Base.SHA1, String}}, ::Any, ::Any) (22 children)
                 175: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.REPLCompletions.get_value(::Expr, ::Module) (22 children)
                 176: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{Char, Any}, ::Any, ::Any) (23 children)
                 177: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{<:Base.Order.ForwardOrdering, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}, ::Int64, ::Int64) (30 children)
                 178: signature Tuple{typeof(!), Any} triggered MethodInstance for isinf(::AbstractFloat) (30 children)
                 179: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::String, ::Any) (31 children)
                 180: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Base.StackTraces.StackFrame}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (37 children)
                 181: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{_A, Vector{Float64}} where _A<:Base.Order.Ordering, ::Int64, ::Int64) (40 children)
                 182: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{String, Any}, ::Any, ::Any) (42 children)
                 183: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEdit.var"#add_nested_key!#24"(::Any, ::typeof(REPL.LineEdit.add_nested_key!), ::Dict, ::Union{Char, String}, ::Any) (47 children)
                 184: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Order.lt(::Base.Order.Perm{<:Base.Order.ReverseOrdering{Base.Order.ForwardOrdering}, <:Union{AbstractVector{Union{Missing, Float32}}, AbstractVector{Union{Missing, Float64}}, AbstractVector{Missing}, AbstractVector{Float32}, AbstractVector{Float64}}}, ::Int64, ::Int64) (64 children)
                 185: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict{_A, Nothing} where _A, ::Nothing, ::Any) (70 children)  
                 186: signature Tuple{typeof(!), Any} triggered MethodInstance for sort!(::Vector{String}, ::Int64, ::Int64, ::Base.Sort.InsertionSortAlg, ::Base.Order.Ordering) (126 children)
                 187: signature Tuple{typeof(!), Any} triggered MethodInstance for setindex!(::Dict, ::Any, ::Any) (254 children)
                 188: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for (::ReverseDiff.SkipOptimize{typeof(!=)})(::Int64, ::ReverseDiff.TrackedReal) (1 children)
                 189: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Artifacts.var"#load_overrides#1"(::Bool, ::typeof(Artifacts.load_overrides)) (1 children)
                 190: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for ReverseDiff.remove_tp(::Expr) (1 children)
                 191: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Artifacts.var"#artifact_meta#12"(::Base.BinaryPlatforms.Platform, ::typeof(Artifacts.artifact_meta), ::String, ::Dict{String, Any}, ::String) (1 children)
                 192: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Artifacts.process_overrides(::Dict{String, Any}, ::Base.UUID) (1 children)
                 193: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for !=(::Int64, ::ForwardDiff.Dual{Ty}) where Ty (1 children)
                 194: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for ReverseDiff.combinations(::Vector{Symbol}, ::Int64) (1 children)
                 195: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for ReverseDiff.remove_tp(::Any) (1 children)
                 196: signature !=(x, y) in Base at operators.jl:282 (formerly !=(x, y) in Base at operators.jl:282) triggered MethodInstance for Base.CoreLogging.logging_error(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Bool) (1 children)
false

julia> show(trees[end-2])
inserting promote_rule(::Type{R}, ::Type{ForwardDiff.Dual{T, V, N}}) where {R<:Real, T, V, N} in ForwardDiff at C:\Users\accou\.julia\packages\ForwardDiff\pDtsf\src\dual.jl:425 invalidated:
   backedges: 1: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{Int64}, ::Type{S} where S<:Real) (5 children)
              2: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{UInt16}, ::Type) (11 children)
              3: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{UInt8}, ::Type) (149 children)
              4: superseding promote_rule(::Type, ::Type) in Base at promotion.jl:310 with MethodInstance for promote_rule(::Type{Int64}, ::Type) (372 children)
   19 mt_cache
false

julia> show(trees[end-3])
inserting tail(t::ChainRulesCore.Tangent{<:NamedTuple{<:Any, <:Tuple{}}}) in ChainRulesCore at C:\Users\accou\.julia\packages\ChainRulesCore\ctmSK\src\tangent_types\tangent.jl:110 invalidated:
   mt_backedges:  1: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base._cshp(::Int64, ::Tuple{Bool}, ::Tuple{Int64}, ::Any) (0 children)
                  2: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base._cshp(::Int64, ::Tuple{Bool}, ::Tuple{Any, Vararg{Any}}, ::Any) (0 children)
                  3: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple, ::Any) (0 children)
                  4: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children)
                  5: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Any, Vararg{Any}}, ::Missing) (0 children)
                  6: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Bool, Vararg{Any}}, ::Bool) (0 children)
                  7: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children)
                  8: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for iterate(::Base.Iterators.Enumerate{Vector{VersionNumber}}, ::Any) (0 children)
                  9: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for VSCodeServer.JuliaInterpreter.optimize!(::Core.CodeInfo, ::Method) (0 children)
                 10: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for JuliaInterpreter.optimize!(::Core.CodeInfo, ::Method) (0 children)
                 11: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                 12: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.tail(::NamedTuple{names}) where names (450 children)
false

julia> show(trees[end-4])
inserting checkindex(::Type{Bool}, inds::OffsetArrays.IdOffsetRange, i::Real) in OffsetArrays at C:\Users\accou\.julia\packages\OffsetArrays\80Lkc\src\axes.jl:276 invalidated:
   mt_backedges:  1: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  2: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  3: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, Type{ColorTypes.RGB{FixedPointNumbers.N0f8}}, Tuple{Vector{ColorTypes.LCHab{Float64}}}}) (0 children)    
                  4: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, Type{ColorTypes.RGB{FixedPointNumbers.N0f8}}, Tuple{Vector{ColorTypes.LCHab{Float64}}}}) (0 children)  
                  5: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  6: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for findall(::Function, ::Vector{Core.LineInfoNode}) (0 children)
                  7: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  8: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for LoweredCodeUtils.step_through_methoddef(::Any, ::JuliaInterpreter.Frame, ::Any) (0 children)
                  9: signature Tuple{typeof(checkindex), Type{Bool}, Any, Int64} triggered MethodInstance for Base.checkbounds_indices(::Type{Bool}, ::Tuple{Any}, ::Tuple{Vararg{Int64}}) (17 children)
                 10: signature Tuple{typeof(checkindex), Type{Bool}, Any, Integer} triggered MethodInstance for Base.checkbounds_indices(::Type{Bool}, ::Tuple, ::Tuple{Integer}) (109 children)
   backedges: 1: superseding checkindex(::Type{Bool}, inds::AbstractUnitRange, i::Real) in Base at abstractarray.jl:727 with MethodInstance for checkindex(::Type{Bool}, ::AbstractUnitRange, ::Int64) (5 children)
              2: superseding checkindex(::Type{Bool}, inds::AbstractUnitRange, i::Real) in Base at abstractarray.jl:727 with MethodInstance for checkindex(::Type{Bool}, ::AbstractUnitRange, ::Integer) (5 children)

[ChrisRackauckas]

Base.promote_rule(::Type{R}, ::Type{Dual{T,V,N}}) where {R<:$R,T,V,N} = Dual{T,promote_type(R, V),N}

@timholy any ideas for anything that can be done for that invalidation?

[ChrisRackauckas]

SciML/DiffEqBase.jl#736 and SciML/OrdinaryDiffEq.jl#1627 together lead to startup times from 8 seconds to 0.7 without a system image, and down to 0.1 and below with a system image.

Without a system image:

using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.847580 seconds (83.25 k allocations: 3.404 MiB, 99.75% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.701598 seconds (499.23 k allocations: 28.846 MiB, 99.73% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.000113 seconds (457 allocations: 39.828 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000147 seconds (950 allocations: 45.547 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23())
# 8.587653 seconds (24.77 M allocations: 3.581 GiB, 5.37% gc time, 99.99% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 1.122847 seconds (3.69 M allocations: 211.491 MiB, 2.45% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23())
# 0.000120 seconds (455 allocations: 39.531 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000138 seconds (950 allocations: 45.188 KiB)

With a system image:

using PackageCompiler
create_sysimage(["OrdinaryDiffEq"], sysimage_path="DiffEqSysImage.so")



using OrdinaryDiffEq
function f(du, u, p, t)
    du[1] = 0.2u[1]
    du[2] = 0.4u[2]
end
u0 = ones(2)
tspan = (0.0, 1.0)
prob = ODEProblem{true,false}(f, u0, tspan, Float64[])

function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
lorenzprob = ODEProblem{true,false}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
typeof(prob) === typeof(lorenzprob) # true

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.133316 seconds (4.66 k allocations: 309.428 KiB, 2.40% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.097989 seconds (432.99 k allocations: 26.587 MiB, 99.01% compilation time)

@time sol = solve(lorenzprob, Rosenbrock23())
# 0.000105 seconds (457 allocations: 39.828 KiB)

@time sol = solve(lorenzprob, Rosenbrock23(autodiff=false))
# 0.000138 seconds (950 allocations: 45.547 KiB)

lorenzprob2 = ODEProblem(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])

@time sol = solve(lorenzprob2, Rosenbrock23())
# 7.651692 seconds (23.73 M allocations: 3.516 GiB, 8.12% gc time, 99.99% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 1.120606 seconds (2.67 M allocations: 145.677 MiB, 27.06% gc time, 99.98% compilation time)

@time sol = solve(lorenzprob2, Rosenbrock23())
# 0.000098 seconds (455 allocations: 39.531 KiB)

@time sol = solve(lorenzprob2, Rosenbrock23(autodiff=false))
# 0.000140 seconds (950 allocations: 45.188 KiB)

Before:

After:

More interestingly, it puts us into the paradigm where most ODE solve calls are actually fully precompiled, separately from the f call.

[ChrisRackauckas]

The function specialization precompilation system is now fully documented:

SciML/SciMLBase.jl#257

https://scimlbase.sciml.ai/stable/interfaces/Problems/#Specialization-Levels

[ChrisRackauckas]

Our goal is to get compile times of all standard workflows to at least 0.1 seconds.

On v1.10 out of the box with no system image:

using OrdinaryDiffEq
function lorenz(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end

@time begin
    lorenzprob = ODEProblem{true, SciMLBase.AutoSpecialize}(lorenz, [1.0; 0.0; 0.0], (0.0, 1.0), Float64[])
    sol = solve(lorenzprob, Rosenbrock23())
end

# 0.096844 seconds (106.87 k allocations: 7.286 MiB, 99.59% compilation time)

Load times can improve still, but I think we can call this one a victory and close it, leaving any specific compile time issues to their own thread.

The details for how this was done is captured in this blog post: https://sciml.ai/news/2022/09/21/compile_time/