API

[pkofod]

So, method basically works now, so I think it might be worth it to start thinking API already.

If this is going to fit in some general "Constrained Optimization" in Optim, or in relation to Optim at least, then a proper API is worth thinking about. How will this fit into an ecosystem where there are also bound constraints? Or other general inequality constraints. There was some work at JuliaNLSolvers/Optim.jl#50 wrt a primal interior point method with inequality constraints. There, a Constraints type was created that collected box constraints, linear constraints, and nonlinear constraints. I will have a look at MathProgBase and see how stuff is organized there.

I think we should aim at having a Constraints type (or a couple of them) different from the *DifferentiableFunction-interface.

[cortner]

I propose ConstrainedOptim.jl and move the AL method in there asap (or just rename this repository and move it to JuliaOpt). As a first step, having equality and bound constraints only allows us to model arbitrary inequality constraints. That would be my recommendation. Bound constraints are already implemented in Optim.jl so this should be straightforward. The algorithm in ALMethod is in fact formulated with bound constraints, so it should be easy to generalise it.

But once this is done and we have at least one method running, then I suggest to get some input from some heavyweights in the optimisation community what the current front-runners are in constrained optimisation - I wouldn't rush this.

Re interface:

• I'm ok with a separate type for constraints. Even if it is the same thing, it will make it easier to read the code.
• Initially I suggest to not treat linear and affine constraints differently. They would anyhow need specialised methods to take advantage. In the long run I'd be interested in trying out optimisation methods that work directly on the manifold and linear constraints would be a trivial case of those.

[pkofod]

I agree, with basically everything. I don't think I have authority to rename the package.

[cortner]

shall I just transfer ownership to you? or to JuliaOpt ?

[pkofod]

I think it is fine to keep it "where it is", we can simply rename it.

I see that we can use fminbox to do the bounds constraints, and here a common dispatch based treatment of constraint types would be helpful. Maybe it would be a good first step to experiment with this in ConstrainedOptim, that is we create a copy of the logarithmic barrier method here.

[cortner]

Has anybody in the Optim community tested how log-barrier methods compare with projection methods? Box constraints are somehow a special case where I believe more efficient schemes are available. An example is the TRON code, but surely there are many others. It is another one of those things where I think we need to just get something up and running and not worry about the best possible solution, but then get some genuine expert advise. When we get to that point I'll have a chat with Nick Gould about it, but I think we also ought to talk to Dominique Orban (@dpo) since he is actually involved in developing Julia codes for optimisation.

[dpo]

I seem to recall that some years ago, while visiting ANL, Nick ran a comparison between TRON and a standard interior-point method for bound-constrained problems (that he wrote I believe). I also seem to recall that his conclusion was that they were performing similarly. He might still have the detailed results or the code he used, I'm not sure. Active-set and interior methods are often used for different purposes however. For instance, including an interior-method as subsolver in a constrained solver is wasteful because of their inability to hot-start. That's where active-set methods shine. We have a PR for TRON over at Optimize.jl, so please feel free to chime in over there.

Speaking broadly for vast categories of problems, the "front-runners" are SQP and interior-point methods but they're very tricky to get right, especially for ill-posed or nearly ill-posed problems. For instance, IPOPT and KNITRO were developed over more than a decade to get to where they are. Is IPOPT not an option for you?

Don't hesitate if there are other questions.

[cortner]

Hi Dominique - thank you for the comments. This is very helpful.

I think everybody here agrees that normally one should use established software. I like to fine-tune though since I target objectives that take hours and sometimes days to evaluate. So a factor 2-3 (even at the expense of robustness) makes a huge difference. Optim.jl is perfect for that.

[pkofod]

Is IPOPT not an option for you?

Make no mistake, no one is naive enough to believe that anything developed here will be cutting edge. Total development time is probably less than 5 hours on average per week, so it wont even be possible to catch up.

That said, I think (Constrained)Optim has it's use in the Julia ecosystem. Everything is close to what you can find in the underlying paper or text book, so that makes the software easy to 1) use to learn from and 2) hack to suit your specific needs. Sure 2) also comes with the chance of doing something stupid, but at least you have the chance to do it if you really want to.