-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Clean up security groups and permissions for extra mounts (#246)
Create a CDK script to automate the creation of security groups for external login nodes and for external FSx file systems. Add a parameter, AdditionalSecurityGroupsStackName to get the security group ids from the created stack and configure the head and compute node additional security groups. Update docs. Update deployment-prerequisites.md. Add security-groups.md. Replace RESEnvironmentName parameter with RESStackName. Get the RESEnvironment from the parameters of the RES stack. Delete SubmitterInstanceTags parameter because not used anywhere. Will add a new parameter to use configure/deconfigure external login nodes. Don't add extramount security groups to parallelcluster. Don't add extra mount security groups to create cluster lambda Update permissions to lambda that creates ParallelCluster. Add ec2:DeleteTags permission Add missing fsx permissions. Use cluster-manager instead of vdc-controller to create users/groups json. Add errors to SNS notification in CreateBuildFiles lambda. Handle special case where the same cluster name exists in multiple VPCs. This causes Route53 hosted zones with the same names and the a record for the head node gets created in the wrong hosted zone. Make sure to send SNS notification if parallelCluster create or update fails.
- Loading branch information
Showing
25 changed files
with
1,045 additions
and
585 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| #!/bin/bash -xe | ||
|
|
||
| cd create-slurm-security-groups | ||
|
|
||
| python3 -m venv .venv | ||
| source .venv/bin/activate | ||
| python3 -m pip install -r requirements.txt | ||
| pwd | ||
| ./create-slurm-security-groups.py "$@" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| *.swp | ||
| package-lock.json | ||
| .pytest_cache | ||
| *.egg-info | ||
|
|
||
| # Byte-compiled / optimized / DLL files | ||
| __pycache__/ | ||
| *.py[cod] | ||
| *$py.class | ||
|
|
||
| # Environments | ||
| .env | ||
| .venv | ||
| env/ | ||
| venv/ | ||
| ENV/ | ||
| env.bak/ | ||
| venv.bak/ | ||
|
|
||
| # CDK Context & Staging files | ||
| .cdk.staging/ | ||
| cdk.out/ | ||
|
|
||
| cdk.context.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
|
|
||
| # Welcome to your CDK Python project! | ||
|
|
||
| You should explore the contents of this project. It demonstrates a CDK app with an instance of a stack (`create_security_groups_stack`) | ||
| which contains an Amazon SQS queue that is subscribed to an Amazon SNS topic. | ||
|
|
||
| The `cdk.json` file tells the CDK Toolkit how to execute your app. | ||
|
|
||
| This project is set up like a standard Python project. The initialization process also creates | ||
| a virtualenv within this project, stored under the .venv directory. To create the virtualenv | ||
| it assumes that there is a `python3` executable in your path with access to the `venv` package. | ||
| If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv | ||
| manually once the init process completes. | ||
|
|
||
| To manually create a virtualenv on MacOS and Linux: | ||
|
|
||
| ``` | ||
| $ python3 -m venv .venv | ||
| ``` | ||
|
|
||
| After the init process completes and the virtualenv is created, you can use the following | ||
| step to activate your virtualenv. | ||
|
|
||
| ``` | ||
| $ source .venv/bin/activate | ||
| ``` | ||
|
|
||
| If you are a Windows platform, you would activate the virtualenv like this: | ||
|
|
||
| ``` | ||
| % .venv\Scripts\activate.bat | ||
| ``` | ||
|
|
||
| Once the virtualenv is activated, you can install the required dependencies. | ||
|
|
||
| ``` | ||
| $ pip install -r requirements.txt | ||
| ``` | ||
|
|
||
| At this point you can now synthesize the CloudFormation template for this code. | ||
|
|
||
| ``` | ||
| $ cdk synth | ||
| ``` | ||
|
|
||
| You can now begin exploring the source code, contained in the hello directory. | ||
| There is also a very trivial test included that can be run like this: | ||
|
|
||
| ``` | ||
| $ pytest | ||
| ``` | ||
|
|
||
| To add additional dependencies, for example other CDK libraries, just add to | ||
| your requirements.txt file and rerun the `pip install -r requirements.txt` | ||
| command. | ||
|
|
||
| ## Useful commands | ||
|
|
||
| * `cdk ls` list all stacks in the app | ||
| * `cdk synth` emits the synthesized CloudFormation template | ||
| * `cdk deploy` deploy this stack to your default AWS account/region | ||
| * `cdk diff` compare deployed stack with current state | ||
| * `cdk docs` open CDK documentation | ||
|
|
||
| Enjoy! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| import aws_cdk as cdk | ||
| from aws_cdk import App, Environment | ||
| from create_slurm_security_groups.create_slurm_security_groups_stack import CreateSlurmSecurityGroupsStack | ||
|
|
||
| app = cdk.App() | ||
|
|
||
| cdk_env = Environment( | ||
| account = app.node.try_get_context('account_id'), | ||
| region = app.node.try_get_context('region') | ||
| ) | ||
| stack_name = app.node.try_get_context('stack_name') | ||
|
|
||
| CreateSlurmSecurityGroupsStack(app, stack_name, env=cdk_env, termination_protection = True,) | ||
|
|
||
| app.synth() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| { | ||
| "app": "python3 app.py", | ||
| "watch": { | ||
| "include": [ | ||
| "**" | ||
| ], | ||
| "exclude": [ | ||
| "README.md", | ||
| "cdk*.json", | ||
| "requirements*.txt", | ||
| "source.bat", | ||
| "**/__init__.py", | ||
| "python/__pycache__", | ||
| "tests" | ||
| ] | ||
| }, | ||
| "context": { | ||
| "@aws-cdk/aws-lambda:recognizeLayerVersion": true, | ||
| "@aws-cdk/core:checkSecretUsage": true, | ||
| "@aws-cdk/core:target-partitions": [ | ||
| "aws", | ||
| "aws-cn" | ||
| ], | ||
| "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true, | ||
| "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true, | ||
| "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true, | ||
| "@aws-cdk/aws-iam:minimizePolicies": true, | ||
| "@aws-cdk/core:validateSnapshotRemovalPolicy": true, | ||
| "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true, | ||
| "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true, | ||
| "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true, | ||
| "@aws-cdk/aws-apigateway:disableCloudWatchRole": true, | ||
| "@aws-cdk/core:enablePartitionLiterals": true, | ||
| "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true, | ||
| "@aws-cdk/aws-iam:standardizedServicePrincipals": true, | ||
| "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true, | ||
| "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true, | ||
| "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true, | ||
| "@aws-cdk/aws-route53-patters:useCertificate": true, | ||
| "@aws-cdk/customresources:installLatestAwsSdkDefault": false, | ||
| "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true, | ||
| "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true, | ||
| "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true, | ||
| "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true, | ||
| "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true, | ||
| "@aws-cdk/aws-redshift:columnId": true, | ||
| "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true, | ||
| "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true, | ||
| "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true, | ||
| "@aws-cdk/aws-kms:aliasNameRef": true, | ||
| "@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true, | ||
| "@aws-cdk/core:includePrefixInUniqueNameGeneration": true, | ||
| "@aws-cdk/aws-efs:denyAnonymousAccess": true, | ||
| "@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true, | ||
| "@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true, | ||
| "@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true, | ||
| "@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true, | ||
| "@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true, | ||
| "@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true, | ||
| "@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true | ||
| } | ||
| } |
Oops, something went wrong.