working on developer ergonomics

[ityonemo]

I'm leaving this issue here to notate that there's some work being done in an experimental branch of annex.

A few things that Jason and I talked about @ elixirconf:

• can we convert the operational semantic into the "pipe".
• how to make functions represent (potentially mutable) weights
• what is the right semantic to communicate and/or patch in computational interfaces, e.g. SIMD, GPU,

A few things that I think are very important for deep learning operator ergonomics

• reifying the concepts of minibatches and batches into the language structure. It's a common problem that corner cases in your pipeline (e.g. batches that have the wrong size because the number of valid samples is wrong, corrupt one-off images, etc.) result in VM crash in Python, Julia, etc. which kills the VM and can potentially make you lose progress. This is infuriating because you "won't know till you've hit it", so if we can bake some of these things into the type system we might be able to get some wins.

Hypothetical code (xor problem, neural transfer):

# these layers are not going to change.
fully_connected_1 = Annex.hydrate("fully_connected.dat", immutable: true)
relu_layer = Annex.new(Annex.Layers.Relu, nodes: 8)

# so let's group them into the transfer component
def evaluate_front_end(data) do
  data
  |> fully_connected_1.()
  |> relu_layer.()
end

# we're only going to train the following layer:
penultimate_layer = Annex.new(Annex.Layers.FullyConnected, input: 8, output: 2)
eval_layer = Annex.new(Annex.Layers.SoftMax, nodes: 2)

[_, newly_trained, _] = Annex.train([&evaluate_front_end/1, penultimate_layer, eval_layer], dataset, cost_function, training_algorithm)

better_result = some_data
|> evaluate_front_end
|> newly_trained
|> eval_layer