Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker: data dir & user permissions, install ca-certificates, better data dir location, more. #58

Merged
merged 13 commits into from
Apr 20, 2023
8 changes: 7 additions & 1 deletion .github/workflows/goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,17 @@ jobs:
- name: Setup Docker Buildx (docker multi-arch dependency)
uses: docker/setup-buildx-action@v2

- name: dockerhub-login
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this deploy to docker hub every time a release tag is created? I'm not familiar with the frequency that this workflow is triggered.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's been triggered once so far.

uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- uses: goreleaser/goreleaser-action@v4
with:
distribution: goreleaser
version: latest
args: release --clean --skip-docker
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Expand Down
4 changes: 4 additions & 0 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ dockers:
- --label=org.opencontainers.image.created={{ .Date }}
- --label=org.opencontainers.image.revision={{ .FullCommit }}
- --label=org.opencontainers.image.licenses=MIT
extra_files:
- docker/docker-entrypoint.sh
- use: buildx
goos: linux
goarch: arm64
Expand All @@ -55,6 +57,8 @@ dockers:
- --label=org.opencontainers.image.created={{ .Date }}
- --label=org.opencontainers.image.revision={{ .FullCommit }}
- --label=org.opencontainers.image.licenses=MIT
extra_files:
- docker/docker-entrypoint.sh

# automatically select amd64/arm64 when requesting "algorand/conduit"
docker_manifests:
Expand Down
22 changes: 14 additions & 8 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,14 +1,20 @@
# This dockerfile is used by goreleaser
FROM debian:bullseye-slim

RUN useradd conduit
RUN mkdir -p /conduit/data && \
chown -R conduit.conduit /conduit
RUN groupadd --gid=999 --system algorand && \
algochoi marked this conversation as resolved.
Show resolved Hide resolved
useradd --uid=999 --no-log-init --create-home --system --gid algorand algorand && \
mkdir -p /data && \
chown -R algorand.algorand /data && \
apt-get update && \
apt-get install -y gosu ca-certificates && \
update-ca-certificates && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

# binary is passed into the build
COPY conduit /conduit/conduit
COPY conduit /usr/local/bin/conduit
COPY docker/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

USER conduit
WORKDIR /conduit
ENTRYPOINT ["./conduit"]
CMD ["-d", "data"]
ENV CONDUIT_DATA_DIR /data
WORKDIR /data
ENTRYPOINT ["docker-entrypoint.sh"]
14 changes: 14 additions & 0 deletions docker/docker-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -e

# To allow mounting the data directory we need to change permissions
# to our algorand user. The script is initially run as the root user
# in order to change permissions, afterwards the script is re-launched
winder marked this conversation as resolved.
Show resolved Hide resolved
# as the algorand user.
if [ "$(id -u)" = '0' ]; then
chown -R algorand:algorand $CONDUIT_DATA_DIR
exec gosu algorand "$0" "$@"
fi

# always run the conduit command
exec conduit "$@"
16 changes: 13 additions & 3 deletions docs/Docker.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
**This container is a work in progress and not yet deployed to docker hub.**

# Docker Image

Algorand's Conduit data pipeline packaged for docker.
Expand Down Expand Up @@ -45,5 +43,17 @@ docker run algorand/conduit init --importer algod --processors filter_processor
With `conduit.yml` in your current working directory,
launch the container:
```
docker run -it -v $(pwd)/conduit.yml:/conduit/data/conduit.yml algorand/conduit
docker run -it -v $(pwd)/conduit.yml:/data/conduit.yml algorand/conduit
```

# Mounting the Data Directory
tzaffi marked this conversation as resolved.
Show resolved Hide resolved

The data directory is located at `/algod/data`. Mounting a volume at that location will allow you to resume the deployment from another container.

## Volume Permissions

The container executes in the context of the `algorand` user with UID=999 and GID=999 which is handled differently depending on your operating system or deployment platform. During startup the container temporarily runs as root in order to modify the permissions of /data. It then changes to the `algorand` user. This can sometimes cause problems, for example if your deployment platform doesn't allow containers to run as the root user.
winder marked this conversation as resolved.
Show resolved Hide resolved

### Use specific UID and GID

On the host system, ensure the directory being mounted uses UID=999 and GID=999. If the directory already has these permissions you may override the default user with `-u 999:999`.
bbroder-algo marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 0 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -84,16 +84,13 @@ github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk5
github.com/algorand/avm-abi v0.1.1/go.mod h1:+CgwM46dithy850bpTeHh9MC99zpn2Snirb3QTl2O/g=
github.com/algorand/avm-abi v0.2.0 h1:bkjsG+BOEcxUcnGSALLosmltE0JZdg+ZisXKx0UDX2k=
github.com/algorand/avm-abi v0.2.0/go.mod h1:+CgwM46dithy850bpTeHh9MC99zpn2Snirb3QTl2O/g=
github.com/algorand/go-algorand-sdk/v2 v2.0.0-20230228201805-5b8c99b1412c h1:KAX6gb3+DLCTBcVhjDtuhcdbCeKnwIYKdj5Dv2JA/nI=
github.com/algorand/go-algorand-sdk/v2 v2.0.0-20230228201805-5b8c99b1412c/go.mod h1:Nt3EHpP8AznLs0/EFfhr0/xsVf5ucnvjNeRygGgbUzM=
github.com/algorand/go-algorand-sdk/v2 v2.0.0-20230324200319-055c8d2b174a h1:fv15GJlyepaaP517PeiJuPX0Q1Wmr17T8uZzevep/TU=
github.com/algorand/go-algorand-sdk/v2 v2.0.0-20230324200319-055c8d2b174a/go.mod h1:Nt3EHpP8AznLs0/EFfhr0/xsVf5ucnvjNeRygGgbUzM=
github.com/algorand/go-codec v1.1.8 h1:XDSreeeZY8gMst6Edz4RBkl08/DGMJOeHYkoXL2B7wI=
github.com/algorand/go-codec v1.1.8/go.mod h1:XhzVs6VVyWMLu6cApb9/192gBjGRVGm5cX5j203Heg4=
github.com/algorand/go-codec/codec v1.1.8 h1:lsFuhcOH2LiEhpBH3BVUUkdevVmwCRyvb7FCAAPeY6U=
github.com/algorand/go-codec/codec v1.1.8/go.mod h1:tQ3zAJ6ijTps6V+wp8KsGDnPC2uhHVC7ANyrtkIY0bA=
github.com/algorand/indexer v0.0.0-20230306212826-146c4d38c5b4 h1:BLzw/1gSbntKblR4ywXdSSxTM/GeKhdkchXNtKUUnzs=
github.com/algorand/indexer v0.0.0-20230306212826-146c4d38c5b4/go.mod h1:ULZ8Qt539rs+FNkSYdoe9HuZ/z1cRAFsWCysylz0nDg=
github.com/algorand/indexer v0.0.0-20230315150109-cf0074cfd4ed h1:aZ5FURJNLUmyayj10ahbVuPJtFQ6YBdp0mP3zJz7yyY=
github.com/algorand/indexer v0.0.0-20230315150109-cf0074cfd4ed/go.mod h1:ULZ8Qt539rs+FNkSYdoe9HuZ/z1cRAFsWCysylz0nDg=
github.com/algorand/oapi-codegen v1.12.0-algorand.0 h1:W9PvED+wAJc+9EeXPONnA+0zE9UhynEqoDs4OgAxKhk=
Expand Down