-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge PR #1234: Cloud network setup scripts
* Remotenet command for AWS and ansible fixes for all remotenet commands * Ansible for AWS, terraform for AWS * Moved remotenet commands to networks/ folder, created ansible playbooks and example scripts to set up remote testnets in the cloud * Added application deployment infrastructure scripts * Obsoleted DigitalOcean scripts, some cleanup in AWS scripts * Changelog -> pending, disclaimer
- Loading branch information
1 parent
671d849
commit 75eeaad
Showing
61 changed files
with
3,282 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| ######################################## | ||
| ### These targets were broken out of the main Makefile to enable easy setup of testnets. | ||
| ### They use a form of terraform + ansible to build full nodes in AWS. | ||
| ### The shell scripts in this folder are example uses of the targets. | ||
|
|
||
| # Name of the testnet. Used in chain-id. | ||
| TESTNET_NAME?=remotenet | ||
|
|
||
| # Name of the servers grouped together for management purposes. Used in tagging the servers in the cloud. | ||
| CLUSTER_NAME?=$(TESTNET_NAME) | ||
|
|
||
| # Number of servers to put in one availability zone in AWS. | ||
| SERVERS?=1 | ||
|
|
||
| # Number of regions to use in AWS. One region usually contains 2-3 availability zones. | ||
| REGION_LIMIT?=1 | ||
|
|
||
| # Path to gaiad for deployment. Must be a Linux binary. | ||
| BINARY?=$(CURDIR)/../build/gaiad | ||
|
|
||
| # Path to the genesis.json and config.toml files to deploy on full nodes. | ||
| GENESISFILE?=$(CURDIR)/../build/genesis.json | ||
| CONFIGFILE?=$(CURDIR)/../build/config.toml | ||
|
|
||
| # Name of application for app deployments | ||
| APP_NAME ?= faucettestnet1 | ||
| # Region to deploy VPC and application in AWS | ||
| REGION ?= us-east-2 | ||
|
|
||
| all: | ||
| @echo "There is no all. Only sum of the ones." | ||
|
|
||
| disclaimer: | ||
| @echo "WARNING: These are example network configuration scripts only and have not undergone security review. They should not be used for production deployments." | ||
|
|
||
| ######################################## | ||
| ### Extract genesis.json and config.toml from a node in a cluster | ||
|
|
||
| extract-config: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b -e TESTNET_NAME="$(TESTNET_NAME)" -e GENESISFILE="$(GENESISFILE)" -e CONFIGFILE="$(CONFIGFILE)" extract-config.yml | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Remote validator nodes using terraform and ansible in AWS | ||
|
|
||
| validators-start: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/terraform-aws && terraform init && (terraform workspace new "$(CLUSTER_NAME)" || terraform workspace select "$(CLUSTER_NAME)") && terraform apply -auto-approve -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" -var TESTNET_NAME="$(CLUSTER_NAME)" -var SERVERS="$(SERVERS)" -var REGION_LIMIT="$(REGION_LIMIT)" | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b -e BINARY=$(BINARY) -e TESTNET_NAME="$(TESTNET_NAME)" setup-validators.yml | ||
| cd remote/ansible && ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b start.yml | ||
|
|
||
| validators-stop: disclaimer | ||
| cd remote/terraform-aws && terraform workspace select "$(CLUSTER_NAME)" && terraform destroy -force -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" && terraform workspace select default && terraform workspace delete "$(CLUSTER_NAME)" | ||
| rm -rf remote/ansible/keys/ remote/ansible/files/ | ||
|
|
||
| validators-status: disclaimer | ||
| cd remote/ansible && ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" status.yml | ||
|
|
||
| #validators-clear: | ||
| # cd remote/ansible && ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b clear-config.yml | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Remote full nodes using terraform and ansible in Amazon AWS | ||
|
|
||
| fullnodes-start: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/terraform-aws && terraform init && (terraform workspace new "$(CLUSTER_NAME)" || terraform workspace select "$(CLUSTER_NAME)") && terraform apply -auto-approve -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" -var TESTNET_NAME="$(CLUSTER_NAME)" -var SERVERS="$(SERVERS)" -var REGION_LIMIT="$(REGION_LIMIT)" | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b -e BINARY=$(BINARY) -e TESTNET_NAME="$(TESTNET_NAME)" -e GENESISFILE="$(GENESISFILE)" -e CONFIGFILE="$(CONFIGFILE)" setup-fullnodes.yml | ||
| cd remote/ansible && ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b start.yml | ||
|
|
||
| fullnodes-stop: disclaimer | ||
| cd remote/terraform-aws && terraform workspace select "$(CLUSTER_NAME)" && terraform destroy -force -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" && terraform workspace select default && terraform workspace delete "$(CLUSTER_NAME)" | ||
| rm -rf remote/ansible/keys/ remote/ansible/files/ | ||
|
|
||
| fullnodes-status: disclaimer | ||
| cd remote/ansible && ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" status.yml | ||
|
|
||
| ######################################## | ||
| ### Other calls | ||
|
|
||
| upgrade-gaiad: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b -e BINARY=$(BINARY) upgrade-gaiad.yml | ||
|
|
||
| list: | ||
| remote/ansible/inventory/ec2.py | python -c 'import json,sys ; print "\n".join(json.loads("".join(sys.stdin.readlines()))["tag_Environment_$(CLUSTER_NAME)"])' | ||
|
|
||
| install-datadog: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| @if [ -z "$(DD_API_KEY)" ]; then echo "DD_API_KEY environment variable not set." ; false ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b -e DD_API_KEY="$(DD_API_KEY)" -e TESTNET_NAME="$(TESTNET_NAME)" -e CLUSTER_NAME="$(CLUSTER_NAME)" install-datadog-agent.yml | ||
|
|
||
| remove-datadog: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(CLUSTER_NAME)" -u centos -b remove-datadog-agent.yml | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Application infrastructure setup | ||
|
|
||
| app-start: disclaimer | ||
| #Make sure you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or your IAM roles set for AWS API access. | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/terraform-app && terraform init && (terraform workspace new "$(APP_NAME)" || terraform workspace select "$(APP_NAME)") && terraform apply -auto-approve -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" -var APP_NAME="$(APP_NAME)" -var SERVERS="$(SERVERS)" -var REGION="$(REGION)" | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(APP_NAME)" -u centos -b -e BINARY=$(BINARY) -e TESTNET_NAME="$(TESTNET_NAME)" -e GENESISFILE="$(GENESISFILE)" -e CONFIGFILE="$(CONFIGFILE)" setup-fullnodes.yml | ||
| cd remote/ansible && ansible-playbook -i inventory/ec2.py -l "tag_Environment_$(APP_NAME)" -u centos -b start.yml | ||
|
|
||
| app-stop: disclaimer | ||
| cd remote/terraform-app && terraform workspace select "$(APP_NAME)" && terraform destroy -force -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" -var APP_NAME=$(APP_NAME) && terraform workspace select default && terraform workspace delete "$(APP_NAME)" | ||
| rm -rf remote/ansible/keys/ remote/ansible/files/ | ||
|
|
||
| # To avoid unintended conflicts with file names, always add to .PHONY | ||
| # unless there is a reason not to. | ||
| # https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html | ||
| .PHONY: all extract-config validators-start validators-stop validators-status fullnodes-start fullnodes-stop fullnodes-status upgrade-gaiad list install-datadog remove-datadog app-start app-stop |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| ######################################## | ||
| ### WARNING: The DigitalOcean scripts are deprecated. They are still here because | ||
| ### they might be useful for developers. | ||
| ### Use -f to call this Makefile: "make -f Makefile.do target" | ||
|
|
||
| # Name of the testnet. Used in chain-id. | ||
| TESTNET_NAME?=remotenet | ||
|
|
||
| # Name of the servers grouped together for management purposes. Used in tagging the servers in the cloud. | ||
| CLUSTER_NAME?=$(TESTNET_NAME) | ||
|
|
||
| # Number of servers deployed in Digital Ocean. | ||
| # Number of servers to put in one availability zone in AWS. | ||
| SERVERS?=1 | ||
|
|
||
| # Path to gaiad for deployment. Must be a Linux binary. | ||
| BINARY?=$(CURDIR)/../build/gaiad | ||
|
|
||
| # Path to the genesis.json and config.toml files to deploy on full nodes. | ||
| GENESISFILE?=$(CURDIR)/../build/genesis.json | ||
| CONFIGFILE?=$(CURDIR)/../build/config.toml | ||
|
|
||
| all: | ||
| @echo "There is no all. Only sum of the ones." | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Extract genesis.json and config.toml from a node in a cluster | ||
|
|
||
| extract-config: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -e TESTNET_NAME="$(TESTNET_NAME)" -e GENESISFILE="$(GENESISFILE)" -e CONFIGFILE="$(CONFIGFILE)" extract-config.yml | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Remote validator nodes using terraform and ansible in Digital Ocean | ||
|
|
||
| validators-start: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/terraform-do && terraform init && (terraform workspace new "$(CLUSTER_NAME)" || terraform workspace select "$(CLUSTER_NAME)") && terraform apply -auto-approve -var DO_API_TOKEN="$(DO_API_TOKEN)" -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" -var TESTNET_NAME="$(CLUSTER_NAME)" -var SERVERS="$(SERVERS)" | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -u root -e BINARY=$(BINARY) -e TESTNET_NAME="$(TESTNET_NAME)" setup-validators.yml | ||
| cd remote/ansible && ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -u root start.yml | ||
|
|
||
| validators-stop: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| cd remote/terraform-do && terraform workspace select "$(CLUSTER_NAME)" && terraform destroy -force -var DO_API_TOKEN="$(DO_API_TOKEN)" -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" && terraform workspace select default && terraform workspace delete "$(CLUSTER_NAME)" | ||
| rm -rf remote/ansible/keys/ | ||
|
|
||
| validators-status: | ||
| cd remote/ansible && ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" status.yml | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Remote full nodes using terraform and ansible in Digital Ocean | ||
|
|
||
| fullnodes-start: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/terraform-do && terraform init && (terraform workspace new "$(CLUSTER_NAME)" || terraform workspace select "$(CLUSTER_NAME)") && terraform apply -var DO_API_TOKEN="$(DO_API_TOKEN)" -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" -var TESTNET_NAME="$(CLUSTER_NAME)" -var SERVERS="$(SERVERS)" | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -e BINARY=$(BINARY) -e TESTNET_NAME="$(TESTNET_NAME)" -e GENESISFILE="$(GENESISFILE)" -e CONFIGFILE="$(CONFIGFILE)" setup-fullnodes.yml | ||
| cd remote/ansible && ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -u root start.yml | ||
|
|
||
| fullnodes-stop: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| cd remote/terraform-do && terraform workspace select "$(CLUSTER_NAME)" && terraform destroy -force -var DO_API_TOKEN="$(DO_API_TOKEN)" -var SSH_PUBLIC_FILE="$(HOME)/.ssh/id_rsa.pub" -var SSH_PRIVATE_FILE="$(HOME)/.ssh/id_rsa" && terraform workspace select default && terraform workspace delete "$(CLUSTER_NAME)" | ||
|
|
||
| fullnodes-status: | ||
| cd remote/ansible && ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" status.yml | ||
|
|
||
|
|
||
| ######################################## | ||
| ### Other calls | ||
|
|
||
| upgrade-gaiad: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi | ||
| @if [ -z "`file $(BINARY) | grep 'ELF 64-bit'`" ]; then echo "Please build a linux binary using 'make build-linux'." ; false ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -e BINARY=$(BINARY) upgrade-gaiad.yml | ||
|
|
||
| list: | ||
| remote/ansible/inventory/digital_ocean.py | python -c 'import json,sys ; print "\n".join(json.loads("".join(sys.stdin.readlines()))["$(CLUSTER_NAME)"]["hosts"])' | ||
|
|
||
| install-datadog: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| @if [ -z "$(DD_API_KEY)" ]; then echo "DD_API_KEY environment variable not set." ; false ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -u root -e DD_API_KEY="$(DD_API_KEY)" -e TESTNET_NAME=$(TESTNET_NAME) -e CLUSTER_NAME=$(CLUSTER_NAME) install-datadog-agent.yml | ||
|
|
||
| remove-datadog: | ||
| @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi | ||
| cd remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l "$(CLUSTER_NAME)" -u root remove-datadog-agent.yml | ||
|
|
||
|
|
||
| # To avoid unintended conflicts with file names, always add to .PHONY | ||
| # unless there is a reason not to. | ||
| # https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html | ||
| .PHONY: all extract-config validators-start validators-stop validators-status fullnodes-start fullnodes-stop fullnodes-status upgrade-gaiad list-do install-datadog remove-datadog | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.