Adjust default values for batching

[gregtatum]

This results in a 2.2x speed up for train-mono tasks according to my testing in #931. There's also a bug with loading in Marian args #933, so I'm putting the values in the decoder.yml file as well.

[ZJaume]

The fp16: true option may be safer than just precision: float16, as it is an alias for float16 precision that also adjusts other precision scaling parameters.

[eu9ene]

One downside of moving it to this config is that some GPUs on Berlin cluster that we plan to include soon don't support half precision decoding. And if I remember correctly it fails silently and produces bad results on decoding, so it's quite dangerous. So, this setting should be likely propagated through marian args or there should be a detection mechanism in the script.

Well, in fact all those values were kind of tuned for those smaller GPUs, so they'll likely OOM. We should either leave the safe values and override them in the production training config template or add a todo and an issue to refactor this to adjust those values based on a GPU model form a python script.

[gregtatum]

I added a TODO for checking that the values are safe #936. The issue is that the decoder.yml is the only place to modify these values, as the config values aren't forwarded, see #933. My CTranslate2 patch stack fixes that piece of it, but I don't want to put it up for review until the experiment results come back.

At the very least in the short term, we should have the config working for our current execution environment.

[gregtatum]

The fp16: true option may be safer than just precision: float16, as it is an alias for float16 precision that also adjusts other precision scaling parameters.

I checked the Marian code and it matters for training, but it's the same for just translating. I'm happy to switch to it though.