Support resource labels for Azure Batch

[bentsherman]

Close #3248

Based on discussions in the issue and with Adam, it seems cost analysis in Azure is tied to pools, and job metadata is not propagated to VMs. As a result, this PR applies resourceLabels to the pool when it is created, and the docs make clear that resource labels in Azure should be defined once for all processes, or else they will be non-deterministic.

Happy to be proven wrong if anyone knows better. But this feature is still useful for pipeline-level labels, like pipeline name, user, project, etc.

@adamrtalbot can you test it?

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	ca8441f
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/64dbb576dfe7eb0008f5791a
😎 Deploy Preview	https://deploy-preview-4178--nextflow-docs-staging.netlify.app/process
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[pditommaso]

This test could be updated

[adamrtalbot]

With config item:

process.resourceLabels = [user: "Adam Talbot", purpose: "test-resource-labels"]

Seems to work:

The metadata isn't applied to the job itself, but we expected this:

[adamrtalbot]

Even better, I changed the config to this:

process.resourceLabels = { [ user: "Adam Talbot", purpose: "test-resource-labels", task: task.process ] }

And now I created a pool for every process in Nextflow, each one with metadata labels for each process:

This means you could tag everything for proper cost evaluation based on tasks.

[bentsherman]

@adamrtalbot what do you think about creating different auto pools for different resource labels? This is how it currently works, if two processes have different resource labels, Nextflow will create two different pools for them.

This would allow the resource labels to be process-specific, but could potentially create lots of otherwise identical pools. I'm wondering if you think it's worth having.

[bentsherman]

My concern is that it could lead to lower resource utilization, since each process would have its own pool rather than many processes sharing one or a few pools, which are more likely to be fully utilized the whole time.

[adamrtalbot]

@adamrtalbot what do you think about creating different auto pools for different resource labels? This is how it currently works, if two processes have different resource labels, Nextflow will create two different pools for them.

Seems perfect.

I will try grouping next, so half the processes go to one pool, half to another, using resourceLabels.

Leaving this setting on was a mistake, because now I have to clear up my mess manually:

deletePoolsOnCompletion   = false

[adamrtalbot]

With new config (pipeline is https://github.com/adamrtalbot/nf-canary/):

process {
    withName: 'TEST_CREATE_FOLDER|TEST_PUBLISH_FILE|TEST_IGNORED_FAIL|TEST_PUBLISH_FOLDER|TEST_CREATE_FILE' {
        resourceLabels = [ user: "Adam Talbot", purpose: "test-resource-labels", group: "A" ]
    }
    withName: 'TEST_SUCCESS|TEST_BIN_SCRIPT|TEST_INPUT|TEST_PASS_FILE|TEST_PASS_FOLDER' {
        resourceLabels = [ user: "Adam Talbot", purpose: "test-resource-labels", group: "B" ]
    }
}

[adamrtalbot]

To expand on why it's perfect, you can effectively use resourceLabels as cost tags now for filtering and management. Let Nextflow handle all the Azure Batch pool management stuff for you and you can check how much you're spending (e.g. per process).

[bentsherman]

Wonderful, then this PR is ready to go