Do not auto-prune instance types if there are too many #235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cartalla
force-pushed
the
220-reducing-number-of-compute-resources-to-aggressively
branch
2 times, most recently
from
1c45cc7 to
d33b6df
Compare
I was previously only allowing 1 memory size/core count combination to keep the number of compute resources down and also was combining multiple instance types in one compute resource if possible. This was to try to maximize the number of instance types that were configured. This led to people not being able to configure the exact instance types they wanted. The preference is to notify the user and let them choose which instances types to exclude or to reduce the number of included types. So, I've reverted to my original strategy of 1 instance type per compute resource and 1 CR per queue. The compute resources can be combined into any queues that the user wants using custom slurm settings. I had to exclude instance types in the default configuration in order to keep from exceeding the PC limits. Resolves #220 Update ParallelCluster version in config files and docs. Clean up security scan.
cartalla
force-pushed
the
220-reducing-number-of-compute-resources-to-aggressively
branch
from
d33b6df to
bb9c752
Compare
cartalla
deleted the
220-reducing-number-of-compute-resources-to-aggressively
branch
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amoutn of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory.
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amoutn of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory.
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Clean up the defaults to be less opinionated. The EDA configuration is opinionated and meant to be a starting point. But the defaults are wide open and will fail because they select too many instance types. So, the errors will give guidance on winnowing down the instance types. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amount of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory. Resolves #261
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Clean up the defaults to be less opinionated. The EDA configuration is opinionated and meant to be a starting point. But the defaults are wide open and will fail because they select too many instance types. So, the errors will give guidance on winnowing down the instance types. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amount of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory. Resolves #261 Add UseOnDemand configuration option that defaults to true so that you could create a cluster with only spot instances. Currently you always had to configure on-demand instances and only spot was optional. Make both purchase options optional, but require at least one.
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Clean up the defaults to be less opinionated. The EDA configuration is opinionated and meant to be a starting point. But the defaults are wide open and will fail because they select too many instance types. So, the errors will give guidance on winnowing down the instance types. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amount of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory. Resolves #261 Add UseOnDemand configuration option that defaults to true so that you could create a cluster with only spot instances. Currently you always had to configure on-demand instances and only spot was optional. Make both purchase options optional, but require at least one.
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Clean up the defaults to be less opinionated. The EDA configuration is opinionated and meant to be a starting point. But the defaults are wide open and will fail because they select too many instance types. So, the errors will give guidance on winnowing down the instance types. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amount of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory. Resolves #261 Add UseOnDemand configuration option that defaults to true so that you could create a cluster with only spot instances. Currently you always had to configure on-demand instances and only spot was optional. Make both purchase options optional, but require at least one.
cartalla
added a commit
that referenced
this pull request
Oct 16, 2024
Default Exclude only works correctly with default includes because it excludes instance types to keep the total instance types down. If user specifies any includes, then the default EDA excludes may exclude instance types that they are trying to include. Only use the default EDA includes and excludes if no includes or excludes are configured. Clean up the defaults to be less opinionated. The EDA configuration is opinionated and meant to be a starting point. But the defaults are wide open and will fail because they select too many instance types. So, the errors will give guidance on winnowing down the instance types. Resolves #262 Restore memory based partitions. Related to #235. Create partitions that include the purchase option (sp|od) and the amount of instance memory. This maintains backward compantibility for those using partitions to select the purchase option and amount of total memory. Resolves #261 Add UseOnDemand configuration option that defaults to true so that you could create a cluster with only spot instances. Currently you always had to configure on-demand instances and only spot was optional. Make both purchase options optional, but require at least one.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was previously only allowing 1 memory size/core count combination to keep the number of compute resources down and also was combining multiple instance types in one compute resource if possible.
This led to people no being able to configure the exact instance types they wanted.
So, I've reverted to my original strategy of 1 instance type per compute resource and 1 CR per queue. The compute resources can be combined into any queues that the user wants using custom slurm settings.
I had to exclude instance types in the default configuration in order to keep from exceeding the PC limits.
Resolves #220
Update ParallelCluster version in config files and docs.
Clean up security scan.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.