"Nadam" and "AMSGrad" are implemented.

[askrix]

"Nadam" and "AMSGrad" are implemented. ;-)

[codecov]

Codecov Report

Merging #450 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #450   +/-   ##
=======================================
  Coverage   92.78%   92.78%           
=======================================
  Files          40       40           
  Lines        1372     1372           
=======================================
  Hits         1273     1273           
  Misses         99       99

Impacted Files	Coverage Δ
src/train.jl	99.06% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ef1357...5e4493c. Read the comment docs.

[oxinabox]

This looks promising, thanks. Nice work diving into a fairly deep part of the system.
Can you add some tests?

@malmaud should probably do the code review, I'm not very familiar with this part of the code myself.

[askrix]

Mr. White,

many thanks for your feedback. I tried my best to get the implementation done. As I wrote you about it in issue #447 , I tested the implementation on my computer using Himmelsbau's function and compared the number of iterations each algorithm needed to converge by the given precision and other parameters.

How should look a test for the algorithms in "TensorFlow.jl"? Should I write for example a linear regression task, where different optimization algorithms are benchmarked against each other?

In connection with my pool request I'd like to ask if it will be possible to reorganize the source code and to export all of the optimization algorithms from "train.jl" in the future. For example by creating an extra sub-folder so that each optimizer will have an own file. (E.g. "Nadam.jl"). I'm asking for it because I'm planning to add doc""" some explanations and notes """ to reference to the publications on which my implementations are based.

Best regards,
Askrix

[askrix]

@malmaud Mr. Malmaud,

so far I still didn't get any further feedback regarding my pull-request. I guess my implementation is either not sufficient good or/and something is missing. Could you tell me what I should correct or/and add to the code?

In the meanwhile I prepared the following optimizers: AdaGrad, Adadelta, RMSProp and AdaMax. Should I open a new pull request?

Best regards,
Askrix

[malmaud]

Sorry for the delay, @askrix. These actually look really good! We need some kind of test, even if it's informal, to make sure these work.

I propose adding tests that run the 'logistic.jl' example, but with your new optimizers instead of the Adam optimizer used now. Then we can at least check that the error goes to near zero. You could just modify logistic.jl to be a function that takes an optimizer constructor as an argument and then modify runtests.jl to pass in all the new optimizers to the logistic example. I'm open to other testing ideas as well.

How does that sound?