Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In Google Batch, support user specified boot images #5268

Merged
merged 3 commits into from
Sep 3, 2024
Merged

In Google Batch, support user specified boot images #5268

merged 3 commits into from
Sep 3, 2024

Conversation

siddharthab
Copy link
Contributor

@siddharthab siddharthab commented Aug 30, 2024

The default boot image is batch-cos, but it may not be desired in all
situations. In particular, there is an ongoing issue in which the
batch-cos image does not retry pulling a docker image if there was a
network issue. The user may also have some some other configuration
pre-configured in their custom boot image.

Signed-off-by: Siddhartha Bagaria [email protected]

@siddharthab siddharthab requested a review from a team as a code owner August 30, 2024 18:41
Copy link

netlify bot commented Aug 30, 2024

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 5fd7618
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/66d6b4fe0f95e600082c5ebc

The default boot image is batch-cos, but it may not be desired in all
situations. In particular, there is an ongoing issue in which the
batch-cos image does not retry pulling a docker image if there was a
network issue. The user may also have some some other configuration
pre-configured in their custom boot image.

Signed-off-by: Siddhartha Bagaria <[email protected]>
Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR. It would be useful to add some unit tests to validate the changes

Signed-off-by: Siddhartha Bagaria <[email protected]>
@siddharthab
Copy link
Contributor Author

Thanks for this PR. It would be useful to add some unit tests to validate the changes

Added a very basic unit test; tested with ./gradlew ':plugins:nf-google:test'. Please let me know if you would like more.

Signed-off-by: Paolo Di Tommaso <[email protected]>
Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all fine. thanks for this contribution

@pditommaso pditommaso merged commit 0aaa648 into nextflow-io:master Sep 3, 2024
22 checks passed
@bentsherman
Copy link
Member

@siddharthab do you know if the boot image can be specified in an instance template? if so, we should add a warning that the boot image option you added will be overridden by the instance template if one is used:

if( task.config.getMachineType()?.startsWith('template://') ) {
if( task.config.getAccelerator() )
log.warn1 'Process directive `accelerator` ignored because an instance template was specified'
if( task.config.getDisk() )
log.warn1 'Process directive `disk` ignored because an instance template was specified'
if( executor.config.cpuPlatform )
log.warn1 'Config option `google.batch.cpuPlatform` ignored because an instance template was specified'
if( executor.config.preemptible )
log.warn1 'Config option `google.batch.premptible` ignored because an instance template was specified'
if( executor.config.spot )
log.warn1 'Config option `google.batch.spot` ignored because an instance template was specified'
instancePolicyOrTemplate
.setInstallGpuDrivers( executor.config.getInstallGpuDrivers() )
.setInstanceTemplate( task.config.getMachineType().minus('template://') )
}

@siddharthab
Copy link
Contributor Author

if so, we should add a warning

You are right. I missed the warnings section. Will send another PR.

pditommaso pushed a commit that referenced this pull request Sep 3, 2024
… [ci fast]

When both boot disk image and instance template are specified, the
instance template gets priority, as with many other options covered by
instance templates. Warn the user in this case.

This was missed in #5268.

Signed-off-by: Siddhartha Bagaria <[email protected]>
pditommaso added a commit that referenced this pull request Sep 4, 2024
The default boot image is batch-cos, but it may not be desired in all
situations. In particular, there is an ongoing issue in which the
batch-cos image does not retry pulling a docker image if there was a
network issue. The user may also have some some other configuration
pre-configured in their custom boot image.

This commit adds the ability to specify a custom boot disk image by using 
the configuration option 

```
google.batch.bootDiskImage = '<NAME>'
```


Signed-off-by: Siddhartha Bagaria <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Co-authored-by: Paolo Di Tommaso <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants