Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Dockerfile to enable running end-to-end tests with docker #1230

Closed
Closed
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
2327bf5
First dockerfile iteration
AgustinRamiroDiaz Aug 5, 2023
883b384
add cargo to path
AgustinRamiroDiaz Aug 5, 2023
357c5f9
Keep improving
AgustinRamiroDiaz Aug 5, 2023
c41193e
closer to finish build
AgustinRamiroDiaz Aug 5, 2023
7112159
All but the test :D
AgustinRamiroDiaz Aug 5, 2023
b10cf5d
add dockerignore
AgustinRamiroDiaz Aug 5, 2023
882a349
Tests run! (they fail but it's something)
AgustinRamiroDiaz Aug 5, 2023
0e4f54c
extract specsheet
AgustinRamiroDiaz Aug 5, 2023
e7f54e2
Reorganize
AgustinRamiroDiaz Aug 5, 2023
9f91cfc
extract more binaries
AgustinRamiroDiaz Aug 5, 2023
75b14c7
extract exa into its own build
AgustinRamiroDiaz Aug 5, 2023
4a5cb90
Remove redundancy
AgustinRamiroDiaz Aug 5, 2023
dec2ddf
fix completion paths
AgustinRamiroDiaz Aug 5, 2023
ba25373
extract kcov
AgustinRamiroDiaz Aug 5, 2023
cf9ebec
Add docs
AgustinRamiroDiaz Aug 7, 2023
98bd231
Fix readme indentation
AgustinRamiroDiaz Aug 8, 2023
244afe7
Remove formatting
AgustinRamiroDiaz Aug 8, 2023
3396883
Mock sudo better
AgustinRamiroDiaz Aug 8, 2023
108ece8
Remove commented out code
AgustinRamiroDiaz Aug 8, 2023
88e3d1c
Improve comment
AgustinRamiroDiaz Aug 8, 2023
d7d2213
Extract repeated image into ARG
AgustinRamiroDiaz Aug 10, 2023
50f8fcc
fix + cache dependencies
AgustinRamiroDiaz Aug 10, 2023
4735044
copy with --link to reduce invalidated layers
AgustinRamiroDiaz Aug 10, 2023
86c74b8
mount cache for apt
AgustinRamiroDiaz Aug 10, 2023
6071758
Use EOF for multiple scripts and resemble more the vagranfile with chmod
AgustinRamiroDiaz Aug 18, 2023
4895ecb
Use heredoc
AgustinRamiroDiaz Aug 18, 2023
9cbb991
remove trailing whitespace
AgustinRamiroDiaz Aug 18, 2023
d45156f
fix build
AgustinRamiroDiaz Aug 19, 2023
59e9c47
explain with comment
AgustinRamiroDiaz Aug 19, 2023
bb31f84
copy source code before symlinks
AgustinRamiroDiaz Aug 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
.git
Dockerfile

# Rust stuff
target

# Vagrant stuff
.vagrant
*.log

# Compiled artifacts
# (see devtools/*-package-for-*.sh)
/exa-linux-x86_64
/exa-linux-x86_64-*.zip
/exa-macos-x86_64
/exa-macos-x86_64-*.zip
/MD5SUMS
/SHA1SUMS

# Snap stuff
parts
prime
stage
*.snap
109 changes: 109 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
FROM rust:1.71.1 AS just
# Install Just, the command runner.
RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/bin

FROM rust:1.71.1 AS specsheet
RUN cargo install -q --git https://github.com/ogham/specsheet

FROM rust:1.71.1 AS cargo-hack
RUN cargo install -q cargo-hack

FROM rust:1.71.1 AS cargo-kcov
RUN cargo install -q cargo-kcov

FROM rust:1.71.1 AS exa
WORKDIR /app
# Copy the source code into the image
COPY . .
# Build exa
RUN cargo build

# We use Ubuntu instead of Debian because the image comes with two-way
# shared folder support by default.
# This image is based from Ubuntu
FROM rust:1.71.1 AS base

# Install the dependencies needed for exa to build
RUN apt-get update -qq
RUN apt-get install -qq -o=Dpkg::Use-Pty=0 \
git gcc curl attr libgit2-dev zip \
fish zsh bash bash-completion

# Install kcov for test coverage
# This doesn’t run coverage over the xtests so it’s less useful for now
COPY --from=cargo-kcov /usr/local/cargo/bin/cargo-kcov /usr/bin/cargo-kcov

RUN apt-get install -y \
cmake g++ pkg-config \
libcurl4-openssl-dev libdw-dev binutils-dev libiberty-dev

RUN ln -s $(which python3) /usr/bin/python
RUN cargo kcov --print-install-kcov-sh | sh

# Create a variety of misc scripts.

RUN ln -sf /vagrant/devtools/dev-run-debug.sh /usr/bin/exa
RUN ln -sf /vagrant/devtools/dev-run-release.sh /usr/bin/rexa

RUN echo -e "#!/bin/sh\ncargo build --manifest-path /vagrant/Cargo.toml \\$@" > /usr/bin/build-exa
RUN ln -sf /usr/bin/build-exa /usr/bin/b

RUN echo -e "#!/bin/sh\ncargo test --manifest-path /vagrant/Cargo.toml \\$@ -- --quiet" > /usr/bin/test-exa
RUN ln -sf /usr/bin/test-exa /usr/bin/t

RUN echo -e "#!/bin/sh\n/vagrant/xtests/run.sh" > /usr/bin/run-xtests
RUN ln -sf /usr/bin/run-xtests /usr/bin/x

RUN echo -e "#!/bin/sh\nbuild-exa && test-exa && run-xtests" > /usr/bin/compile-exa
RUN ln -sf /usr/bin/compile-exa /usr/bin/c

ADD --chmod=+x devtools/dev-package-for-linux.sh /vagrant/devtools/dev-package-for-linux.sh
RUN echo -e "#!/bin/sh\nbash /vagrant/devtools/dev-package-for-linux.sh \\$@" > /usr/bin/package-exa
RUN echo -e "#!/bin/sh\ncat /etc/motd" > /usr/bin/halp
AgustinRamiroDiaz marked this conversation as resolved.
Show resolved Hide resolved

# Configure the welcoming text that gets shown:

# Capture the help text so it gets displayed first
RUN rm -f /etc/update-motd.d/*
ADD --chmod=+x devtools/dev-help.sh /vagrant/devtools/dev-help.sh
RUN bash /vagrant/devtools/dev-help.sh > /etc/motd

# Tell bash to execute a bunch of stuff when a session starts
RUN echo "source /vagrant/devtools/dev-bash.sh" > ${HOME}/.bash_profile

# Link the completion files so they’re “installed”:

# bash
RUN ln -s /vagrant/completions/bash/exa /etc/bash_completion.d/exa
RUN ln -s /vagrant/completions/bash/exa /usr/share/bash-completion/completions/exa

# zsh
RUN ln -s /vagrant/completions/zsh/_exa /usr/share/zsh/vendor-completions/_exa

# fish
RUN ln -s /vagrant/completions/fish/exa.fish /usr/share/fish/completions/exa.fish
AgustinRamiroDiaz marked this conversation as resolved.
Show resolved Hide resolved


# Copy the source code into the image
COPY --chmod=+x . /vagrant/
AgustinRamiroDiaz marked this conversation as resolved.
Show resolved Hide resolved

# Make sudo dummy replacement
# This is needed for some tests that use sudo
RUN echo -e '#!/bin/sh\n"$@"' > /usr/bin/sudo
RUN chmod +x /usr/bin/sudo
Comment on lines +103 to +106

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really shouldn't be necessary to rely on sudo if the intent is to replace Vagrantfile with Dockerfile.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I just don't want to touch all the internal files because a) I'm not too familiar with them and b) I'm not sure it'll behave the same as I couldn't run the Vagrant tests. I think we should go iteratively on this, by first making it compatible and afterwards refactoring the sudo stuff


RUN bash /vagrant/devtools/dev-set-up-environment.sh

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore this

This also has some odd locale logic at the end. I'm wondering if that's necessary?

In docker-mailserver image we have a similar locale section:

RUN <<EOF
  rm -rf /usr/share/locale/*
  rm -rf /usr/share/man/*
  rm -rf /usr/share/doc/*
  update-locale
EOF

For exa, I tracked it down to a specific test for different locales, so ignore this feedback 😅

115315a

RUN bash /vagrant/devtools/dev-create-test-filesystem.sh

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if you can use mknod with the image build to get the /dev special files created:

sudo mknod "$TEST_ROOT/specials/block-device" b 3 60
sudo mknod "$TEST_ROOT/specials/char-device" c 14 40
sudo mknod "$TEST_ROOT/specials/named-pipe" p

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I don't understand. Do you mean that mknod won't work in docker?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't think it works, but I could be mistaken, I haven't checked 🤷‍♂️


WORKDIR /vagrant
COPY --from=exa /app/target /vagrant/target

# TODO: remove this once tests don't depend on it
RUN ln -s /vagrant/* ${HOME}

COPY --from=specsheet /usr/local/cargo/bin/specsheet /usr/bin/specsheet
COPY --from=cargo-hack /usr/local/cargo/bin/cargo-hack /usr/bin/cargo-hack
COPY --from=just /usr/bin/just /usr/bin/just

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kconv should be located down here with the rest? I assume the concern was due to the package install portion for kconv and moving the layer(s) down to the end here being problematic when an earlier instruction invalidates the Dockerfile cache?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could actually shift these up to after kconv, shouldn't be an issue given what the rest of the stage is doing?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kconv should be located down here with the rest?

Actually it's needed for RUN cargo kcov --print-install-kcov-sh | sh here

You could actually shift these up to after kconv, shouldn't be an issue given what the rest of the stage is doing?

That's true, do you think it's useful? I don't understand the benefit

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the benefit

The layers that are more frequently to change should come later. The bulk of this stage is something that could run in a separate script via RUN, but for now you've stated you want to keep it in the Dockerfile.

You could move these COPY lines up before kconv for better co-location, but right now no major concern either way.


FROM base AS test
CMD ["/vagrant/xtests/run.sh"]
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,9 +234,26 @@ The full command is `cargo build --release --target=x86_64-unknown-linux-musl --

For more information, see the [Building from Source page](https://the.exa.website/install/source).

### End to end testing

### Testing with Vagrant
#### Testing with containers

You can use [Docker](https://www.docker.com/) to run exa in a container, and test it against a known state.

```fish

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
```fish
```sh

# Build the container image
docker build . -t exa-test --target test

# Run the tests in the container
docker run -it exa-test

# In case you want to get into the container to test something
docker run -it exa-test fish
```

> Note: You should be able to run this with any container runtime you'd like. You are welcome to add a PR documenting how to do this with other runtimes.

#### Testing with Vagrant
exa uses [Vagrant][] to configure virtual machines for testing.

Programs such as exa that are basically interfaces to the system are [notoriously difficult to test][testing].
Expand Down Expand Up @@ -269,4 +286,4 @@ Once this is done, you can SSH in, and build and test:

Of course, the drawback of having a standard development environment is that you stop noticing bugs that occur outside of it.
For this reason, Vagrant isn’t a *necessary* development step — it’s there if you’d like to use it, but exa still gets used and tested on other platforms.
It can still be built and compiled on any target triple that it supports, VM or no VM, with `cargo build` and `cargo test`.
It can still be built and compiled on any target triple that it supports, VM or no VM, with `cargo build` and `cargo test`.