Skip to content

Latest commit

 

History

History
206 lines (141 loc) · 7.61 KB

CUSTOM_BUILDS.md

File metadata and controls

206 lines (141 loc) · 7.61 KB
Raw

AutoSpotting Setup

It's usually recommended to use the provided binaries available as Docker images, but in some cases you may need to customize AutoSpotting for your own environment.

Docker

Pre-built Docker images for the latest evaluation builds are also available on the Docker Hub at AutoSpotting/AutoSpotting

docker run autospotting/autospotting

They might be useful for quick tests, otherwise you might need to build your own docker images.

The repository contains a Dockerfile and docker-compose configuration that allows you to build AutoSpotting Docker container images and run them conveniently on your local machine without installing the Go build environment usually required for local development(which is also documented below).

This can be useful for trying it out locally or even for running it on a container hosting solution such as Kubernetes. They won't support the full functionality that relies on CloudWatch Events but it's probably enough for some people.

If you have docker and docker-compose installed, it's as simple as running

docker-compose run autospotting

This also accepts all the AutoSpotting command-line arguments, including -help which explains all the other available options.

The usual AWS credential environment variables listed in the docker-compose.yaml configuration file are passed to the running container and will need to be set for it to actually work.

Using your own Docker images in AWS Lambda

AutoSpotting uses a Lambda function configured to use a Docker image. Such a configuration currently requires the Docker image to be stored in an ECR from your own account.

AutoSpotting trunk currently builds and runs by default as ARM binaries. Building it locally most probably requires a docker buildx setup, as per the official Docker documentation. The Marketplace version is currently available only at x86 binaries.

Building it as Intel binaries is still possible, but the infrastructure code will currently expect ARM binaries.

In order to support the AWS Marketplace setup, which relies on an ECR repository hosted in another AWS-managed account, the current CloudFormation template uses a custom resource that copies the Docker image from a source ECR (by default the Marketplace ECR) into an ECR created inside the CloudFormation stack. This adds some complexity but has the nice side effect of being able to push the image to any arbitrary ECR in another account/region, offering more flexibility for customers who may want to manage custom deployments at scale.

You'll therefore need to build an ARM Docker image, upload it to an ECR repository in your AWS account and configure your CloudFormation or Terraform stack to use this new image as a source image.

  1. Set up an ECR repository in your AWS account that will host your custom Docker images.

  2. The build system can use a DOCKER_IMAGE variable that tells it where to upload the image. Set it into your environment to the name of your ECR repository. When unset you'll attempt to push to the Marketplace ECR and you'll receive permission errors.

    export DOCKER_IMAGE=1234567890123.dkr.ecr.<region>.amazonaws.com/<my-ecr-name>
export DOCKER_IMAGE_VERSION=1.0.2 # it's strongly recommended versioning images

  3. Define some AWS credentials or profile information into your environment.

  4. Authenticate to your ECR repository

    make docker-login

  5. Build and upload your Docker image to your ECR and configure a CloudFormation template to use your ECR

    make docker-push-artifacts

  6. Use the CloudFormation template from the build directory to create the resources. Make sure you set the parameters SourceECR and SourceImage to point to your ECR repository (SourceECR should be set to contain the hostname part of your ECR repository, before the first / character and SourceImage should contain the rest). The version number will be set automatically based on the value you defined earlier.

    AutoSpotting should now be running against the binaries you built locally and uploaded to your own ECR repository.

    The same process can be used for updating AutoSpotting to a newer version.

Maintaining your own fork

It is recommended to contribute your changes into the mainline version of the project whenever possible, so that others can benefit from your enhancements and bug fixes, but for some reasons you may still want to run your own fork.

Unfortunately the golang import paths can make it tricky, but there is a nice article which documents the problem in detail and gives a couple of possible workarounds.

Make directives

The Makefile from the root of the git repository contains a number of useful directives, they're not documented here as they might change over time, so you may want to have a look at it.

Local Development setup

AutoSpotting is written in Go so for local development you need a Go toolchain. You can probably also use docker-compose for this to avoid it as mentioned above but I prefer the native Go tooling which offers faster feedback for local development.

Dependencies

  1. Install Go, git, Docker and the AWS command-line tool. You may use the official binaries or your usual package manager, whatever you prefer is fine.

  2. Verify that they were properly installed.

    go version, should be at least 1.7

    git version

    docker version

    aws --version

Compiling the binaries locally

  1. Set up a directory for your Go development. I'm using go in my home directory for this example.

  2. Set the GOPATH environment variable to point at your go directory:

    export GOPATH=$HOME/go

    Optionally add this line to your .bash_profile to persist across console sessions.

  3. Run the following command to install the AutoSpotting project into your GOPATH directory:

    go get github.com/AutoSpotting/AutoSpotting

    This downloads the source from GitHub, pulls in all necessary dependencies, builds it for local execution and deploys the binary into the golang binary directory which you may also want to append to your PATH.

  4. Navigate to the root of the AutoSpotting repository:

    cd $GOPATH/src/github.com/AutoSpotting/AutoSpotting

  5. (Optional) You may want to make a minor change to the source code so you can tell when the tool is running your own custom-built version. If so, add a line like this to the autospotting.go file's main() function:

    fmt.Println("Running <my organization name> binaries")

  6. (Optional) Try building and running the test suite locally to make sure everything works correctly:

    make test

  7. Build the code again:

    make build

Running locally

  1. Run the code, assuming you have AWS credentials defined in your environment or in the default AWS credentials profile:

    ./AutoSpotting

    You may also pass some command line flags, see the --help output for more information on the available options.

    When you are happy with how your custom build behaves, you can generate a

    build for AWS Lambda using the Docker method documented above.

Back to the main Readme