Ensure training pipeline is deterministic

[jackapbutler]

In order to decrease sources of error we want to make sure the training pipeline creates a deterministic ordering of the dataset batches and other elements regarding initialisation. This will help to track down the sources of loss spikes i.e. malformed samples.

See this experiment for more information. We might want to check out this.

[jackapbutler]

In the current non-nightly releases of PyTorch the setting above causes training to hang and is a known issue huggingface/transformers#22363, going to try torch.backends.cudnn.deterministic = True.

[jackapbutler]

Using torch.backends.cudnn.deterministic = True and model.eval() to counteract dropout layers the loss for each of the first N batches became more similar but not completely consistent.