GCP: label resources for better cost controlling

[m-rieke]

New feature

Hi everyone!
For our use within the company we need to be able to separate costs for different runs. The one possibility is doing that over different projects, but we are doing that by using google resource labels.

Usage scenario

(What's the main usage case and the deployment scenario addressed by this proposal)

Suggest implementation

Only few modifications are needed. We just had to implement a new configuration item for "google" executor where the labels can be defined. Afterwards these labels just have to be applied to runPipeline method and to VirtualMachine object in createResources method

[m-rieke]

Just for letting you know: There is a pull request for adding this functionality: #2589

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dougnukem]

The GCP labeling being added #2853 (comment) appears to be done at the Google Lifesciences Pipeline level, for this to apply to the GCP VMs for cost monitoring this needs to be done at the VirtualMachine.labels level.

https://github.com/nextflow-io/nextflow/pull/2853/files#diff-bd0a55afab4a626d773cac6787aef85e2d2756a3a25d438df70bb98ae0127f1fR479

I believe instead this should be applied instead or in addition at the Resource/VirtualMachine level.

nextflow/plugins/nf-google/src/main/nextflow/cloud/google/lifesciences/GoogleLifeSciencesHelper.groovy

Lines 195 to 199 in 58be212

	def vm = new VirtualMachine()
	.setMachineType(req.machineType)
	.setDisks([disk])
	.setServiceAccount(serviceAccount)
	.setPreemptible(req.preemptible)

https://cloud.google.com/life-sciences/docs/reference/rpc/google.cloud.lifesciences.v2beta#runpipelinerequest

RunPipelineRequest
labels | map<string, string>User-defined labels to associate with the returned operation. These labels are not propagated to any Google Cloud Platform resources used by the operation, and can be modified at any time.To associate labels with resources created while executing the operation, see the appropriate resource message (for example, VirtualMachine).

https://cloud.google.com/life-sciences/docs/reference/rpc/google.cloud.lifesciences.v2beta#virtualmachine

VirtualMachine
labels | map<string, string>Optional set of labels to apply to the VM and any attached disk resources. These labels must adhere to the name and value restrictions on VM labels imposed by Compute Engine.Labels keys with the prefix 'google-' are reserved for use by Google.Labels applied at creation time to the VM. Applied on a best-effort basis to attached disk resources shortly after VM creation.

Optional set of labels to apply to the VM and any attached disk resources. These labels must adhere to the name and value restrictions on VM labels imposed by Compute Engine.

Labels keys with the prefix 'google-' are reserved for use by Google.

Labels applied at creation time to the VM. Applied on a best-effort basis to attached disk resources shortly after VM creation.