- In case you are not already, register at Docker Hub (https://hub.docker.com/)
- To make a later step more straight forward, use a password without any special characters.
- From now on we refer to your login ID from Docker Hub as
DOCKER_USERNAME
and your password there is calledDOCKER_PASSWORD
.
- Login at Docker Hub and create three public repositories with the following names (by clicking the big blue
Create Repository
button in the top right).mysqlimage
minitwitimage
flagtoolimage
- You need to be signed up at DigitalOcean (https://www.digitalocean.com/).
- Fork this repository (https://github.com/itu-devops/itu-minitwit-ci) by clicking on the fork button on Github
- Clone your fork of the repository:
$ git clone https://github.com/<you_gh_user>/itu-minitwit-ci.git
$ cd itu-minitwit-ci
-
Add your Docker Hub credentials to the
Vagrantfile
, that is replace<your_dockerhub_id>
and<your_dockerhub_pwd>
on lines 36 and 37 with your login ID and password. OBS: Remember to not push these credentials back to a public repository. -
Install the Travis CLI client, which is provided via a Ruby GEM. Make sure you have at least Ruby 2.3.0 (2.6.0 recommended) installed. You can check your Ruby version by running
ruby -v
.- To install the Travis CLI client on Linux (and MacOS) run
sudo gem install travis --no-document
. - In case you are on a Windows machine run
$ gem install travis
- See the official documentation
- To install the Travis CLI client on Linux (and MacOS) run
-
To allow the Travis CLI tool to log into your Travis CI account later, create a GitHub Personal Access Token, see the official documentation:
- Give the token a name and select the scopes
repo
andadmin:org
- Give the token a name and select the scopes
To setup this scenario we have two parts:
- A remote server to which we will deploy our ITU-MiniTwit application and which is provisioned on DigitalOcean using
vagrant
. - A Travis CI pipeline, which we will use to automate tests, build the application (in Docker images) and deploy them to the server.
To connect to the server we are going to provision, we will use RSA keys for authentication. We will provide the same SSH keys to the Travis CI pipeline, so that it can automatically deploy new versions on our server.
Now, create a keypair as illustrated below. The -m "PEM"
sets the format of the generated key pair to a format that is supported by Travis CI.
$ cd ssh_keys
$ ssh-keygen -m "PEM"
When prompted for the file to save the key (Enter file in which to save the key (/home/<your_user>/.ssh/id_rsa)
) type ./ssh_keys/do_ssh_key
. Hit enter three times to accept the other defaults. You can call the SSH key files whatever you want, but the Vagrantfile
expects the SSH keys to have that specific name. So in case you use another name, adapt the Vagrantfile
accordingly.
Now, after generating the key pair, log into DigitalOcean and navigate to the security configuration, left bottom under ACCOUNT
-> Security
.
Under SSH keys
click the Add SSH Key
button and register a New SSH key
with the name do_ssh_key
. Paste into the input field the contents of ssh_keys/do_ssh_key.pub
, which you might receive via: cat ssh_keys/do_ssh_key.pub
on the command line.
Add the generated SSH key to your Github repository, so that Travis CI can clone the repository when we run the pipeline. Navigate to the page of the repository on github.com in your browser and click the Settings
tab:
Now click Deploy keys
and then Add deploy key
. Set the title to something like "Travis CI pipeline" so that you know who has access through this key. Then paste the contents of the public key of the SSH keys we generated earlier (cat ssh_keys/do_ssh_keys.pub
).
Finish by clicking Add key
.
We assume you have vagrant
and the Vagrant DigitalOcean plugin installed, see session 3 prep. notes.
To create virtual machines at DigitalOcean with vagrant
we must generate an authentication token. If you did so already during the last exercise session, you can skip this section. Otherwise, log into DigitalOcean in your browser, then navigate to API
in the menu on the right, then click on Generate New Token
. You must give it a name, for example the name of the machine where you use the token.
The Vagrantfile
expects to find your DigitalOcean token in a respective environment variable in your shell environment. For example, you can add it to your ~/.bashrc
or ~/.zshrc
. The variable must be called: called DIGITAL_OCEAN_TOKEN
, the syntax for defining such an environment variable in your shell configuration file is:
export DIGITAL_OCEAN_TOKEN=<your-token>
After adding the token you must reload your shell. Either close your current terminal and open a new one or use the source
command on the shell config file you changed, e.g., source ~/.bashrc
.
Now, you should be able to create the remote VM via vagrant up
. You can use the below command to ensure that vagrant will use the DigitalOcean provider:
$ vagrant up --provider=digital_ocean
Note down the IP of this server as we will need it in a later step. It should be displayed after the server was created.
All files contained in the directory remote_files
will be synced to the newly provisioned server. Currently, this is only a single docker-compose.yml
file, which later will be used to deploy our ITU-MiniTwit application automatically.
If you need to SSH to remote server you can easily do it through vagrant
with the ssh
command:
$ vagrant ssh
You can also do it 'manually' like so:
$ ssh root@<digital-ocean-machine-ip> -i <path_to/do_ssh_key>
Now, we will setup the Travis CI pipeline.
Start by signing up for Travis CI. Navigate with your browser to https://travis-ci.com
and select Sign up with Github
.
Then, you will be taken to the authorization page, confirm the use of the app.
You should now be redirected to an empty dashboard.
Now, we will setup Travis CI to run a CI/CD pipeline whenever we push commits to our repository. Start by clicking on your profile picture in the upper right corner (make sure to click the image and not the dropdown arrow!) and click Settings
.
Now, we must authorize the "Github Apps Integration". Click the Activate Button
.
Then approve.
Now, you should see a list of your repositories. Use the search bar to filter to for the itu-minitwit-ci
repository and click its name:
For this scenario, we have to share the SSH key in an appropriate format so that Travis CI can clone this repository to build it. We need the travis
CLI tool to add the SSH key to build from a public repository.
Log into your Travis CI account from the terminal with the travis
CLI tool (for reproducing this scenario, remember the --com
option, since Travis has two hosting domains).
travis login --com --github-token <YOUR-TOKEN>
Once you've been authorized (message like Successfully logged in as <your_gh_username>!
), encrypt your SSH key (remember the -r
option, so Travis knows which repository you are reffering to):
travis encrypt-file ssh_keys/do_ssh_key --com -r <YOUR-USER>/itu-minitwit-ci
The above command creates an outpu similar to the following:
encrypting ssh_keys/do_ssh_key for <your_user>/itu-minitwit-ci
storing result as do_ssh_key.enc
storing secure env variables for decryption
Please add the following to your build script (before_install stage in your .travis.yml, for instance):
openssl aes-256-cbc -K $encrypted_<key_value>_key -iv $encrypted_<iv_value>_iv -in do_ssh_key.enc -out ssh_keys/do_ssh_key -d
Pro Tip: You can add it automatically by running with --add.
Make sure to add do_ssh_key.enc to the git repository.
Make sure not to add ssh_keys/do_ssh_key to the git repository.
Commit all changes to your .travis.yml.
Afterwards, take the values from <key_value>
and from <iv_value>
from the above output and store them in the .travis.yml
file (both marked with <REPLACE_ME>
in the .travis.yml
file that you pulled).
Now, add do_ssh_key.enc
to version control (git add ssh_keys/do_ssh_key.enc
, git commit ...
, git push
, ...).
Do the same with the modified .travis.yml
file.
Remember to commit and push the two updated files to your fork of the repository.
Next, we have to add some environment variables, scroll to Environment Variables
, which you can reach under the Settings of the build job for this repository.
For this scenario you must set the following environment variables:
DOCKER_USERNAME
username for hub.docker.comDOCKER_PASSWORD
password for username for hub.docker.comMT_USER
the user we will SSH to, default isroot
MT_SERVER
the IP address of the server we created on DigitalOcean, which you noted down earlier.
These are key-value pairs that are substitutes for their actual value when the pipeline runs. They are never printed to any logs, so this is the way to add "secrets" to your pipeline, like login usernames and passwords.
In order to build using Travis CI pipelines, we must add a file to the root of the Github repository called .travis.yml
that contains the all of the commands to be executed by the pipeline. The nice thing about this being a file in our Git repository is that we can version it along with the rest of our code and keep all of our code and configuration in the same place without having to use any web GUI's - Configuration as Code!
The scenario should already have a sample .travis.yml
in the repository.
os: linux
dist: bionic
language: python
python:
- 3.7
services:
- docker # required, but travis uses older version of docker :(
before_install:
# Decrypt the git_deploy_key.enc key into /tmp/git_deploy_key
- openssl aes-256-cbc -K $encrypted_<REPLACE_ME>_key -iv $encrypted_<REPLACE_ME>_iv -in do_ssh_key.enc -out /tmp/git_deploy_key -d
# Make sure only the current user can read the private key
- chmod 600 /tmp/git_deploy_key
# Create a script to return the passphrase environment variable to ssh-add
- echo 'echo ${SSH_PASSPHRASE}' > /tmp/askpass && chmod +x /tmp/askpass
# Start the authentication agent
- eval "$(ssh-agent -s)"
# Add the key to the authentication agent
- DISPLAY=":0.0" SSH_ASKPASS="/tmp/askpass" setsid ssh-add /tmp/git_deploy_key </dev/null
install:
- docker --version # document the version travis is using
stages:
- docker_build
- test
- deploy
jobs:
include:
- stage: docker_build
name: "build and push docker"
script:
- echo "LOGIN"
- echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
- echo "BUILD"
- docker build -t $DOCKER_USERNAME/minitwitimage:latest . -f Dockerfile-minitwit
- docker build -t $DOCKER_USERNAME/mysqlimage:latest . -f Dockerfile-mysql
- docker build -t $DOCKER_USERNAME/flagtoolimage:latest . -f Dockerfile-flagtool
- echo "PUSH"
- docker push $DOCKER_USERNAME/minitwitimage:latest
- docker push $DOCKER_USERNAME/mysqlimage:latest
- docker push $DOCKER_USERNAME/flagtoolimage:latest
- stage: test
name: "run pytest"
install: skip
script:
- docker build -t $DOCKER_USERNAME/minitwittestimage -f Dockerfile-minitwit-tests .
- yes | docker-compose up -d
- docker run -it --rm --network=itu-minitwit-network $DOCKER_USERNAME/minitwittestimage
- stage: deploy
name: "deploy new version"
install: skip
# -o flag to get around "add ip to known hosts prompt"
script: |
ssh -o "StrictHostKeyChecking no" ${MT_USER}@${MT_SERVER} \
"source /root/.bash_profile && \
cd /vagrant && \
docker-compose pull && \
docker-compose up -d && \
docker pull $DOCKER_USERNAME/flagtoolimage:latest"
This pipeline is divided into three stages:
docker_build
- The build stage will first build our three Docker containers and subsequently push them to
hub.docker.com
.
- The build stage will first build our three Docker containers and subsequently push them to
test
- The test stage runs some automated frontend tests on our application, to check that everything is still working. If the test fails the pipeline will abort and alert you that the tests are failing.
deploy
- The final stage deploys the new version to our remote server by opening an SSH connection and, which remotely sets-up the environment variables (
source /root/.bash_profile
), pulls the freshly built Docker images from hub.docker.com (docker-compose pull
), and finally updates the running containers to the new version (docker-compose up -d
).
- The final stage deploys the new version to our remote server by opening an SSH connection and, which remotely sets-up the environment variables (
Note, that each stage is executed in a freshly provisioned VM, so no state carries over from stage to stage, unless you explicitly tell Travis CI to do so.
Now we are ready to trigger the pipeline. If all of the above went well, a new version of ITU-MiniTwit should be build, tested, delivered, and deployed on every new commit to the repository.
This scenario exists only due to the hard work of the TAs:
For some more details on the docker images see the file readme_dockerized.md