Explore requirements for model retraining pipeline

[stevehadd]

This is likely to be something we try as part of our PRISM scenario, so we should start thinking about the scientific and technical requirements for a pipeline to do this. Things to consider are:

• how does the pipeline differ from training from scratch (cold start) vs retraining (warm start)?
• How might we change hyperparameter in such a scenario (e.g. learning rate) so it doesn't overfit/underfit either the new data or the old data?
• How can we select data for retraining ?
• How do we compare performance and measure gains from the retraining, if there are any? (probably several models available in parallel in. "parallel suite" sort of paradigm.