[docs] update autodiff tutorial

[gdalle]

Two modifications to the tutorial on autodiff of user-defined operators:

1. Upgrade to Enzyme v0.13 (no need for autodiff_deferred anymore). I didn't necessarily go for the most efficient solutions, you may want to ask @wsmoses for that, I just performed the minimal changes to gain compatibility with the breaking release.
2. Add a section demonstrating DifferentiationInterface. For some reason it doesn't work with AutoEnzyme() but it's a start.

Related issues:

• Where is JuMP's sparse AD code, and can DifferentiationInterface help? #3806
• Suggestions for documentation improvements #2348 (comment)

[gdalle]

I had a question about the use of DI. For many backends, performance is greatly improved by the preparation mechanism, which allows you to record a tape / preallocate a cache / etc. But to apply it, you need an example input that has the same type and size as the ones you'll encounter during optimization. Is there any way we can use this mechanism in JuMP?

[odow]

You could just use the first call to build the tape, and then error if there are calls from different inputs. JuMP doesn't provide anything special. You can do everything in user-land.

[wsmoses]

Perhaps add a disclaimer:

While DI makes it easier to experiment with different autodiff tools, this comes at a cost. In particular, DI may produce slower derivatives than using an AD tool directly, and sometimes fail to work at all. For example, in this code passing AutoEnzyme to DI fails to differentiate the code which is supported when directly calling Enzyme, like above.

[gdalle]

By the way do you have any idea why it fails here? As usual it's a segfault so not particularly user friendly

[wsmoses]

No idea, I haven’t even tried to run it or seen what the error message, just took your word that it failed xD.

open an issue with the mwe and error and can take a look

[odow]

and sometimes fail to work at all

I don't want to mention something in our documentation if this is the case.

What is the failure mode? Will we gracefully report errors? Segfaults are not good.

[wsmoses]

Oh agreed in that segfaults are bad, in that case @gdalle just drop DI docs then?

[odow]

I realized my changes mean we never actually mention Enzyme in the context of DI, so perhaps that's okay.

[wsmoses]

The fact that it is a segfault is not due to DI per se, it's Enzyme not catching the error gracefully. Sure, DI might be using Enzyme wrong in this particular case (perhaps because of a function returning two closures), but I too would very much prefer a clean error message from Enzyme to a segfault. Will try to reproduce tomorrow.

Again haven't seen the error message so not clear what's up, but I'm unaware of any segfaults caused by Enzyme atm. Most the time people saw segfaults in the past were actually bugs in the Julia compiler itself (e.g. JuliaLang/julia#55306 ).

That said unfortunately even if you're not the underlying cause of an error, the top level package you use is going to be the one people blame for issues (especially if code works otherwise without it). Analogously, there's a ton of code in Enzyme working around bugs in the Julia and LLVM compiler errors =/. Perhaps something similar would be good in DI?

[gdalle]

It is not DI's job to work around LLVM errors, it's Enzyme's or Julia's. DI is a fancy argument-passer

[gdalle]

And here's the MWE: EnzymeAD/Enzyme.jl#1942

[gdalle]

For the record, pure Enzyme also fails on this one when one uses splat(f)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.90%. Comparing base (288dca8) to head (23c82e8).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3836   +/-   ##
=======================================
  Coverage   97.90%   97.90%           
=======================================
  Files          44       44           
  Lines        6019     6019           
=======================================
  Hits         5893     5893           
  Misses        126      126           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.