Enable stages to request GPU resources

[AlexIoannides]

Description
As a Machine Learning Engineer, I would like to be able to request GPU resources for stages in a workflow, so that tensor-based machine learning (e.g. PyTorch), can benefit from hardware based acceleration for training and serving.

Tasks

• manual PoC based on resources listed below.
• add a gpu_request config parameter for all stage types.
• extend bodywork.k8s.batch_jobs and bodywork.k8s.service_deployments to request GPU resources.
• think of a functional test - e.g. trying to run a deployment that uses PyTorch, explicitly checking for GPU availability and printing to stdout?
• update docs accordingly.

Resources

• Scheduling GPUs
• GPU Support for eksctl
• GPUs on AKS