abstract DifferentiableFunction

[alyst]

This is related to #53, #94, #102 etc.
With the current implementation of DifferentiableFunction it's not very convenient to implement functions that require external data for evaluation. It always ends up writing something like

df = DifferentiableFunction(x -> real_func(x, data) .... )

before every optimizer invocation.

This PR turns DifferentiableFunction into an abstract type, and calls like df.f(x) are supposed to be replaced by evalf(df, x).
The old DifferentiableFunction is renamed into SimpleDifferentiableFunction. Since SimpleDifferentiableFunction is used when one wants to convert 3 functions into DifferentiableFunction object, the current Optim code does not require any modifications.

[johnmyleswhite]

I'm not really sold on this change. In general, I think we should exploit attempts to optimize closures in Base Julia rather than try to work around it.

I also would like to see us move away from the DifferentiableFunction type someday and use normal function types.

[alyst]

It might be regarded as a closure performance workaround, but the intention is to allow callable objects be used with Optim package.
I thought subclassing DifferentiableFunction would be also a natural way to specify how the gradient/hessian should be calculated, i.e. Autodiff, FiniteDifference (subtype allows specifying whether we want left, center or right) etc.

[johnmyleswhite]

I'm hoping to move in the opposite direction as we redesign the API, so closing out.

[alyst]

Is there some document that outlines the new design of Optim?
I would like to understand better how it fits the use cases I have in mind.

[johnmyleswhite]

In the long term, I'd like to move towards an API that takes in arbitrary callables and uses only the number of arguments and the types of the trailing arguments to do dispatch.

For a concrete example, something like optimize(f, x0::Vector, ::Type{LBFGS}) vs optimize(f, g!, x0::Vector, ::Type{LBFGS}) would be used instead of the differentiable function machinery. We would have implicit contracts on f and g! rather than trying to force them via DifferentiableFunction.

We could also adopt a more complex API like that used by NLopt. For now, I mostly want to remove the gap between the optimize wrapper and the inner methods that get called because the optimize wrapper gets tasked with too much moving around of arguments.

[timholy]

In the long term, I'd like to move towards an API that takes in arbitrary callables and uses only the number of arguments and the types of the trailing arguments to do dispatch.

I think this PR implements the very API you're talking about. evalfg!(df, x, gr) is a much more general interface (one that relies on dispatch) than df.fg!(x, gr). #163 provides another example of where you need to be able to pass in (and dispatch on) something that's not one of the "core" numerical types.

Reopen?