nnzj in NLPModels

[aldma]

Dear JSOptimizers,
I encountered an issue while using SlackModel, which possibly traces back to the use or definition of nnzj, lin_nnzj and nln_nnzj in NLPModelsMeta.

It should be reproducible with

using NLPModels, NLPModelsModifiers, CUTEst
nlp = CUTEstModel("HS75")
jac(nlp, nlp.meta.x0)
snlp = SlackModel(nlp)
jac(snlp, snlp.meta.x0)

Using julia 1.10.4, CUTEst v0.13.2, NLPModels v0.20.0, NLPModelsModifiers v0.7.2, I have experienced the following error

BoundsError: attempt to access 14-element Vector{Int64} at index [11:22]

Looking closely at the HS75 problem and the nlp.meta structure, we have
nlp.meta.nnzj = 12
nlp.meta.lin_nnzj = 8
nlp.meta.nln_nnzj = 12
and somewhat surprisingly NLPModels.jac manages to return the Jacobian matrix. There are indeed 12 nonzeros but only 4 for linear constraints and 8 nonlinear ones.

• I would think that nnzj = lin_nnzj + nln_nnzj, but it doesn't seem to be the case. What do these properties mean?

For the SlackModel snlp we have
snlp.meta.nnzj = 14
snlp.meta.lin_nnzj = 10
snlp.meta.nln_nnzj = 12
after two slack variables are introduced (correctly). In this case, however, NLPModels.jac fails with the error shown above.

• Do you have any idea how to fix this?

Thank you!

[aldma]

For the same nlp = CUTEstModel("HS75") I observe that

julia> jac_lin_structure(nlp)
([1, 1, 2, 2, 1, 1, 1, 1], [1, 2, 2, 1, 1, 1, 1, 1])

julia> jac_nln_structure(nlp)
([1, 1, 1, 2, 2, 2, 3, 3, 0, 0, 0, 0], [2, 1, 4, 2, 1, 3, 1, 2, 0, 0, 0, 0])

The first one has repeated entry pairs, but still works, whereas the second assigns zeros. Going into jac_structure! of the NLPModels API, this makes it crash at Line 194, when reading nlp.meta.nln[rows[i]] with rows[i]=0.

[dpo]

Thank you for the report @aldma. Indeed, it seems like a bug for zeros to appear in the sparsity structure. If I understand well, you are using nlp, which means that the message above is what happens before you add the slack variables. Correct?

[dpo]

I remember a recent conversation with @tmigot about those zeros. I can reproduce the issue above. I don’t think they should be there. This behavior:

ulia> jac_structure(nlp)
([1, 1, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5], [1, 2, 2, 1, 2, 1, 4, 2, 1, 3, 1, 2])

julia> jac_lin_structure(nlp)
([1, 1, 2, 2, 1, 1, 1, 1], [1, 2, 2, 1, 1, 1, 1, 1])

julia> jac_nln_structure(nlp)
([1, 1, 1, 2, 2, 2, 3, 3, 0, 0, 0, 0], [2, 1, 4, 2, 1, 3, 1, 2, 0, 0, 0, 0])

julia> nlp.meta.nnzj, nlp.meta.lin_nnzj, nlp.meta.nln_nnzj
(12, 8, 12)

doesn’t make much sense. @tmigot Could you remind me what the logic was for those zeros?

[dpo]

I don’t see why the lines https://github.com/JuliaSmoothOptimizers/CUTEst.jl/blob/main/src/julia_interface.jl#L453 are there. I would expect nnzj = lin_nnzj + nln_nnzj. Is it because we can’t predict that there will be duplicate entries?

[tmigot]

You are right to assume that nnzj = lin_nnzj + nln_nnzj (see JuliaSmoothOptimizers/NLPModels.jl#448) this is just a bug in the implementation.

[tmigot]

Hi guys! I think the issue come from the way we initialize lin_nnzj and nln_nnzj in CUTEst.jl:

CUTEst.jl/src/CUTEst.jl

Line 383 in f2ad6a4

lin_nnzj = min(nvar * nlin, nnzj)

There should be a better way.

(moving this issue to CUTEst.jl)

[MaxenceGollier]

I have a similar issue on the problem HS42,
when I write

using NLPModels, CUTEst
nlp = CUTEstModel("HS42")

I get the error message

AssertionError: nnzj == lin_nnzj + nln_nnzj

This is due to JuliaSmoothOptimizers/NLPModels.jl#448, the initialization of lin_nnzj should be different as mentionned by @tmigot

[amontoison]

I asked Nick how we can determine lin_nnzj: ralna/CUTEst#59

[amontoison]

It should be fixed with #351.
The next release will be v0.14.0.