Precompilation failure on v1.8 (related to MUMPS_seq_jll?)

[thchr]

I'm having trouble precompiling this package on Julia v1.8 (everything works fine on v1.7); my troubles seem related to MUMPS_seq_jll.

Specifically, if I try to add Cbc (in a completely clean package environment), precompilation fails for several jll dependencies of Cbc:

(@v1.8) pkg> add Cbc
[...]
Precompiling project...
  ✗ Clp_jll
  ✗ Cgl_jll
  ✗ Cbc_jll
  ✗ Cbc
  43 dependencies successfully precompiled in 50 seconds
  4 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages

Additionally MUMPS_seq_jll is marked as outdated:

(@v1.8) pkg> st --outdated -m
Status `C:\Users\tchr\.julia\environments\v1.8\Manifest.toml`
⌅ [d7ed1dd3] MUMPS_seq_jll v5.5.1+0 (<v400.1000.0+0): Clp_jll
⌅ [656ef2d0] OpenBLAS32_jll v0.3.17+0 (<v0.3.20+0): julia

The versions of Cbc, Clp_jll, Cgl_jll, and Cbc_jll are:

  [9961bab8] Cbc v1.0.1
  [38041ee0] Cbc_jll v200.1000.501+0
  [3830e938] Cgl_jll v0.6000.300+0
  [06985876] Clp_jll v100.1700.601+0

Full stack trace from precompilation below:

Precompilation stack trace

julia> import Pkg; Pkg.precompile()
Precompiling project...
  ✗ Clp_jll
  ✗ Cgl_jll
  ✗ Cbc_jll
  ✗ Cbc
  0 dependencies successfully precompiled in 10 seconds. 43 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

Cbc [9961bab8-2fa3-5c5a-9d89-47fab24efd76]

Failed to precompile Cbc [9961bab8-2fa3-5c5a-9d89-47fab24efd76] to C:\Users\tchr\.julia\compiled\v1.8\Cbc\jl_DDCF.tmp.
ERROR: LoadError: InitError: could not load library "C:\Users\tchr\.julia\artifacts\029cea9e5bf8cca60ec7023a09ff1c9a45091f9a\bin\libcmumps.dll"
The specified module could not be found.
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl .\libdl.jl:117
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl .\libdl.jl:116
  [3] macro expansion
    @ C:\Users\tchr\.julia\packages\JLLWrappers\QpMQW\src\products\library_generators.jl:54 [inlined]
  [4] __init__()
    @ MUMPS_seq_jll C:\Users\tchr\.julia\packages\MUMPS_seq_jll\ziXhU\src\wrappers\x86_64-w64-mingw32-libgfortran5.jl:17
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base .\loading.jl:831
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base .\loading.jl:1039
  [7] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1315
  [8] _require_prelocked(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1200
  [9] macro expansion
    @ .\loading.jl:1180 [inlined]
 [10] macro expansion
    @ .\lock.jl:223 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1144
 [12] include(mod::Module, _path::String)
    @ Base .\Base.jl:419
 [13] top-level scope
    @ C:\Users\tchr\.julia\packages\JLLWrappers\QpMQW\src\toplevel_generators.jl:188
 [14] include
    @ .\Base.jl:419 [inlined]
 [15] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base .\loading.jl:1554
 [16] top-level scope
    @ stdin:1
during initialization of module MUMPS_seq_jll
in expression starting at C:\Users\tchr\.julia\packages\Clp_jll\5lq5B\src\wrappers\x86_64-w64-mingw32-libgfortran5-cxx11.jl:7
in expression starting at C:\Users\tchr\.julia\packages\Clp_jll\5lq5B\src\Clp_jll.jl:2
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Clp_jll [06985876-5285-5a41-9fcb-8948a742cc53] to C:\Users\tchr\.julia\compiled\v1.8\Clp_jll\jl_E1B3.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base .\loading.jl:1705
  [3] compilecache
    @ .\loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1200
  [6] macro expansion
    @ .\loading.jl:1180 [inlined]
  [7] macro expansion
    @ .\lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1144
  [9] include(mod::Module, _path::String)
    @ Base .\Base.jl:419
 [10] top-level scope
    @ C:\Users\tchr\.julia\packages\JLLWrappers\QpMQW\src\toplevel_generators.jl:188
 [11] include
    @ .\Base.jl:419 [inlined]
 [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base .\loading.jl:1554
 [13] top-level scope
    @ stdin:1
in expression starting at C:\Users\tchr\.julia\packages\Cgl_jll\yAgOf\src\wrappers\x86_64-w64-mingw32-cxx11.jl:4
in expression starting at C:\Users\tchr\.julia\packages\Cgl_jll\yAgOf\src\Cgl_jll.jl:2
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Cgl_jll [3830e938-1dd0-5f3e-8b8e-b3ee43226782] to C:\Users\tchr\.julia\compiled\v1.8\Cgl_jll\jl_E02C.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base .\loading.jl:1705
  [3] compilecache
    @ .\loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1200
  [6] macro expansion
    @ .\loading.jl:1180 [inlined]
  [7] macro expansion
    @ .\lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1144
  [9] include(mod::Module, _path::String)
    @ Base .\Base.jl:419
 [10] top-level scope
    @ C:\Users\tchr\.julia\packages\JLLWrappers\QpMQW\src\toplevel_generators.jl:188
 [11] include
    @ .\Base.jl:419 [inlined]
 [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base .\loading.jl:1554
 [13] top-level scope
    @ stdin:1
in expression starting at C:\Users\tchr\.julia\packages\Cbc_jll\1RLJl\src\wrappers\x86_64-w64-mingw32-libgfortran5-cxx11.jl:5
in expression starting at C:\Users\tchr\.julia\packages\Cbc_jll\1RLJl\src\Cbc_jll.jl:2
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Cbc_jll [38041ee0-ae04-5750-a4d2-bb4d0d83d27d] to C:\Users\tchr\.julia\compiled\v1.8\Cbc_jll\jl_DE48.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base .\loading.jl:1705
  [3] compilecache
    @ .\loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1200
  [6] macro expansion
    @ .\loading.jl:1180 [inlined]
  [7] macro expansion
    @ .\lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1144
  [9] include
    @ .\Base.jl:419 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1554
 [11] top-level scope
    @ stdin:1
in expression starting at C:\Users\tchr\.julia\packages\Cbc\ncnr8\src\Cbc.jl:6
in expression starting at stdin:1
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types C:\Users\tchr\.julia\juliaup\julia-1.8.0+0.x64\share\julia\stdlib\v1.8\Pkg\src\Types.jl:67
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API C:\Users\tchr\.julia\juliaup\julia-1.8.0+0.x64\share\julia\stdlib\v1.8\Pkg\src\API.jl:1427
 [3] precompile
   @ C:\Users\tchr\.julia\juliaup\julia-1.8.0+0.x64\share\julia\stdlib\v1.8\Pkg\src\API.jl:1058 [inlined]
 [4] #precompile#225
   @ C:\Users\tchr\.julia\juliaup\julia-1.8.0+0.x64\share\julia\stdlib\v1.8\Pkg\src\API.jl:1057 [inlined]
 [5] precompile (repeats 2 times)
   @ C:\Users\tchr\.julia\juliaup\julia-1.8.0+0.x64\share\julia\stdlib\v1.8\Pkg\src\API.jl:1057 [inlined]
 [6] top-level scope
   @ REPL[2]:1

---

On the other hand, if I first install MUMPS_seq_jll manually, precompilation succeeds:

(@v1.8) pkg> add MUMPS_seq_jll
(@v1.8) pkg> add Cbc
(@v1.8) pkg> st
Status `C:\Users\tchr\.julia\environments\v1.8\Project.toml`
  [9961bab8] Cbc v1.0.1
  [d7ed1dd3] MUMPS_seq_jll v400.1000.0+0

But this causes the previously failing Cbc jll dependencies to be marked as outdated:

(@v1.8) pkg> status --outdated -m
Status `C:\Users\tchr\.julia\environments\v1.8\Manifest.toml`
⌃ [38041ee0] Cbc_jll v2.10.5+2 (<v200.1000.501+0)
⌅ [3830e938] Cgl_jll v0.60.2+6 (<v0.6000.300+0): Cbc_jll
⌅ [06985876] Clp_jll v1.17.6+7 (<v100.1700.601+0): Cbc_jll, Cgl_jll
⌅ [be027038] CoinUtils_jll v2.11.3+4 (<v200.1100.400+0): Cbc_jll, Clp_jll, Osi_jll
⌅ [656ef2d0] OpenBLAS32_jll v0.3.17+0 (<v0.3.20+0): julia
⌅ [7da25872] Osi_jll v0.108.5+4 (<v0.10800.600+0): Cbc_jll, Clp_jll

---

System info (Windows)

julia> versioninfo()
Julia Version 1.8.0
Commit 5544a0fab7 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
  Threads: 1 on 8 virtual cores

[odow]

cc @amontoison here's another MUMPS_seq_jll issue on Julia v1.8.

x-ref: jump-dev/Ipopt.jl#327

For now a work-around is probably

] add Cbc MUMPS_seq_jll@5.4.1

There's some issue with the v5.5.1 version of MUMPS_seq_jll on Windows in Julia 1.8.

[amontoison]

@odow One issue comes from the PR of @blegat JuliaPackaging/Yggdrasil#4806
MUMPS_seq_jll v4.0.3 has only static libraries.
I don't know why it wasn't recompiled with the version number 4.0.4.

[odow]

How does that PR affect [email protected]?

[amontoison]

@blegat
I don't understand why we have a version 401.1000.000 of MUMPS_seq_jll 🤔
It should break all packages that rely on the shared libraries of MUMPS_seq_jll (versions 5.x.x).
JuliaRegistries/General#58881 (comment)

[amontoison]

@odow
Can you open a PR to check if Cbc can be precompiled on Linux / OSX with Julia v1.8?
I need to check if the version 401.1000.000 or 5.5.1 is installed by default.
It seems that Cbc is unable to open / find the shared of MUMPS_seq_jll.

[odow]

Actually, here's one from yesterday: https://github.com/jump-dev/Cbc.jl/runs/8060859973?check_suite_focus=true, which uses MUMPS_seq_jll v5.5.1+0 on Windows and everything is fine.

[odow]

@thchr what happens if you just run ] add [email protected], and then using MUMPS_seq_jll? And what happens if you try the work-around of ] add Cbc [email protected]?

[odow]

I just had a chat to @amontoison and we think we have a patch to fix this issue: JuliaRegistries/General#67310

It's unrelated to Ipopt.

[thchr]

@thchr what happens if you just run ] add [email protected], and then using MUMPS_seq_jll?

That throws on the using MUMPS_seq_jll with:

Stack trace

```jl julia> using MUMPS_seq_jll ERROR: InitError: could not load library "C:\Users\tchr\.julia\artifacts\029cea9e5bf8cca60ec7023a09ff1c9a45091f9a\bin\libcmumps.dll" The specified module could not be found. Stacktrace: [1] dlopen(s::String, flags::UInt32; throw_error::Bool) @ Base.Libc.Libdl .\libdl.jl:117 [2] dlopen(s::String, flags::UInt32) @ Base.Libc.Libdl .\libdl.jl:116 [3] macro expansion @ C:\Users\tchr\.julia\packages\JLLWrappers\QpMQW\src\products\library_generators.jl:54 [inlined] [4] __init__() @ MUMPS_seq_jll C:\Users\tchr\.julia\packages\MUMPS_seq_jll\ziXhU\src\wrappers\x86_64-w64-mingw32-libgfortran5.jl:17 [5] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any}) @ Base .\loading.jl:831 [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64) @ Base .\loading.jl:1039 [7] _require(pkg::Base.PkgId) @ Base .\loading.jl:1315 [8] _require_prelocked(uuidkey::Base.PkgId) @ Base .\loading.jl:1200 [9] macro expansion @ .\loading.jl:1180 [inlined] [10] macro expansion @ .\lock.jl:223 [inlined] [11] require(into::Module, mod::Symbol) @ Base .\loading.jl:1144 during initialization of module MUMPS_seq_jll ```

And what happens if you try the work-around of ] add Cbc [email protected]?

This works without a hitch :)

Thanks for looking at a fix for this so quickly!

[odow]

Okay, the problem is definitely that MUMPS_seq_jll v5.5.1 is broken on Windows. @amontoison has a plan to recompile things, but the patch to the General registry will prevent people installing Cbc from getting the broken version of MUMPS.

Another work-around is to use HiGHS instead. At this point, I would encourage all users of Cbc to switch to HiGHS.

[odow]

@thchr can you try now in a fresh environment? The registry should be updated.

[thchr]

Works now! Thanks!

I tried the HiGHS solver per your suggestion, but it did not seem generically faster than Cbc for the problems I had (MILP via LayeredLayouts.jl): what's the rationale for switching?

[odow]

what's the rationale for switching?

HiGHS is supported and actively developed.