Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ONBUILD executed in multi-stage build grandchild #5578

Open
MikeMcC399 opened this issue Dec 7, 2024 · 8 comments · May be fixed by #5593 or #5596
Open

ONBUILD executed in multi-stage build grandchild #5578

MikeMcC399 opened this issue Dec 7, 2024 · 8 comments · May be fixed by #5593 or #5596

Comments

@MikeMcC399
Copy link

MikeMcC399 commented Dec 7, 2024

Observed Behavior

If a docker build is executed using the Docker image cypress/factory in a two-stage Dockerfile build, with the recommended

# syntax=docker/dockerfile:1

from Custom Dockerfile syntax then the ONBUILD RUN instructions from cypress/factory are executed correctly in the first build stage, and fail in the second build stage. The ONBUILD instructions can only be executed once, since they clean up after themselves and they delete resources which are no longer supposed to be necessary.

cypress/factory is built with factory/factory.Dockerfile and this includes:

# Global Cleanup
ONBUILD RUN apt-get purge -y --auto-remove \
    curl \
    bzip2 \
    gnupg \
    dirmngr\
  && rm -rf /usr/share/doc \
  && rm -rf /var/lib/apt/lists/* \
  # Remove cypress install scripts
  && rm -rf /opt/installScripts

This is a regression in docker/dockerfile:1.11.0 and above:

docker/dockerfile version Result
undefined success
1 fail
1.10.0 success
1.11.0 fail
1.11.1 fail
1.12.0 fail

Expected Behavior

The set of ONBUILD instructions from an existing Docker file should only be used in the first stage of a multi-stage build, not in the second or any following stages.

Steps to reproduce

See https://github.com/MikeMcC399/cy-docker-multistage

On Ubuntu 24.04.1 LTS, Node.js v22.12.0 LTS, Docker Desktop 4.36.0

git clone https://github.com/MikeMcC399/cy-docker-multistage
cd cy-docker-multistage
npm ci
docker build . -f Dockerfile.1 -t cy-test --no-cache

using Dockerfile.1:

# syntax=docker/dockerfile:1
FROM cypress/factory AS first

COPY . /opt/app
WORKDIR /opt/app
RUN npm install cypress --save-dev
RUN npx cypress install

FROM first AS second

RUN apt-get update
RUN apt-get install chromium -y

Logs

$ docker build . -f Dockerfile.1 -t cy-test
[+] Building 94.7s (20/28)                                                                                                                                                                         docker:default
 => [internal] load build definition from Dockerfile.1                                                                                                                                                       0.1s
 => => transferring dockerfile: 267B                                                                                                                                                                         0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                                                                                                                    1.8s
 => docker-image://docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e76dddbb6d75c4c5                                                                                     2.3s
 => => resolve docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e76dddbb6d75c4c5                                                                                         0.0s
 => => sha256:61782c2b067b8aa6d6a5a44dda352a26cc0bcb2dcf01b45b6eea6fca5a518bba 850B / 850B                                                                                                                   0.0s
 => => sha256:b968edff90bd031526591b03dfbc60323746c2d57ae359d0bff26f2e971b0094 1.26kB / 1.26kB                                                                                                               0.0s
 => => sha256:dae058f2fffd4fce12a0791304a7160b8e92593dd7bb824d0c2aa12f346f9f2b 12.78MB / 12.78MB                                                                                                             2.0s
 => => sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e76dddbb6d75c4c5 8.40kB / 8.40kB                                                                                                               0.0s
 => => extracting sha256:dae058f2fffd4fce12a0791304a7160b8e92593dd7bb824d0c2aa12f346f9f2b                                                                                                                    0.1s
 => [internal] load build definition from Dockerfile.1                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/cypress/factory:latest                                                                                                                                            1.3s
 => [internal] load .dockerignore                                                                                                                                                                            0.1s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => [first 1/5] FROM docker.io/cypress/factory:latest@sha256:ece75e68a6ef4b92c296b1bee68c4c7fcec7e30a2b0aff1394c7b6092aabd450                                                                               31.6s
 => => resolve docker.io/cypress/factory:latest@sha256:ece75e68a6ef4b92c296b1bee68c4c7fcec7e30a2b0aff1394c7b6092aabd450                                                                                      0.0s
 => => sha256:bc0965b23a04fe7f2d9fb20f597008fcf89891de1c705ffc1c80483a1f098e4f 28.23MB / 28.23MB                                                                                                             4.2s
 => => sha256:7e0a9cd7b743bea5b7e3c15257e5a44693f6b7adcfdd21b42349842ca82e5fec 163.09MB / 163.09MB                                                                                                          22.2s
 => => sha256:3dac0345d6caafaef6a8d9e008355a1ce7c6f48919891b30c8042ce22675f903 4.66kB / 4.66kB                                                                                                               0.8s
 => => sha256:ece75e68a6ef4b92c296b1bee68c4c7fcec7e30a2b0aff1394c7b6092aabd450 683B / 683B                                                                                                                   0.0s
 => => sha256:7ba1dee0d47c173e3b04bd88b44062d513adb02b022b63ffe8b181815e68d044 899B / 899B                                                                                                                   0.0s
 => => sha256:2a6063523a72719fcd3ff210bf3f88d5b722caa38e79dffb5015261e775b2b26 3.09kB / 3.09kB                                                                                                               0.0s
 => => extracting sha256:bc0965b23a04fe7f2d9fb20f597008fcf89891de1c705ffc1c80483a1f098e4f                                                                                                                    2.2s
 => => extracting sha256:7e0a9cd7b743bea5b7e3c15257e5a44693f6b7adcfdd21b42349842ca82e5fec                                                                                                                    8.9s
 => => extracting sha256:3dac0345d6caafaef6a8d9e008355a1ce7c6f48919891b30c8042ce22675f903                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                            1.1s
 => => transferring context: 26.98MB                                                                                                                                                                         1.0s
 => [first  2/20] ONBUILD RUN bash /opt/installScripts/node/install-node-version.sh 22.12.0                                                                                                                 16.5s
 => [first  3/20] ONBUILD RUN node /opt/installScripts/yarn/install-yarn-version.js ${YARN_VERSION}                                                                                                          0.4s
 => [first  4/20] ONBUILD RUN node /opt/installScripts/chrome/install-chrome-version.js ${CHROME_VERSION}                                                                                                    0.4s
 => [first  5/20] ONBUILD RUN node /opt/installScripts/edge/install-edge-version.js ${EDGE_VERSION}                                                                                                          0.5s
 => [first  6/20] ONBUILD RUN node /opt/installScripts/firefox/install-firefox-version.js ${FIREFOX_VERSION}                                                                                                 0.5s
 => [first  7/20] ONBUILD RUN node /opt/installScripts/cypress/install-cypress-version.js ${CYPRESS_VERSION}                                                                                                 0.6s
 => [first  8/20] ONBUILD RUN apt-get purge -y --auto-remove     curl     bzip2     gnupg     dirmngr  && rm -rf /usr/share/doc   && rm -rf /var/lib/apt/lists/*   && rm -rf /opt/installScripts             1.6s
 => [first  9/20] COPY . /opt/app                                                                                                                                                                            0.6s
 => [first 10/20] WORKDIR /opt/app                                                                                                                                                                           0.0s
 => [first 11/20] RUN npm install cypress --save-dev                                                                                                                                                         1.6s
 => [first 12/20] RUN npx cypress install                                                                                                                                                                   34.1s
 => ERROR [second  1/17] ONBUILD RUN bash /opt/installScripts/node/install-node-version.sh 22.12.0                                                                                                           0.3s
------
 > [second  1/17] ONBUILD RUN bash /opt/installScripts/node/install-node-version.sh 22.12.0:
0.289 bash: /opt/installScripts/node/install-node-version.sh: No such file or directory
------
Dockerfile.1:9
--------------------
   7 |     RUN npx cypress install
   8 |
   9 | >>> FROM first AS second
  10 |
  11 |     RUN apt-get update
--------------------
ERROR: failed to solve: process "/bin/sh -c bash /opt/installScripts/node/install-node-version.sh ${APPLIED_FACTORY_DEFAULT_NODE_VERSION}" did not complete successfully: exit code: 127

Workaround

Pin syntax to 1.10.0 in Dockerfile.1.10.0:

# syntax=docker/dockerfile:1.10.0

To test workaround, as above, except:

docker build . -f Dockerfile.1.10.0 -t cy-test --no-cache # pin to older version
npm test                                        # run Cypress test

Reference

Dockerfile ONBUILD

Triggers are cleared from the final image after being executed. In other words they aren't inherited by "grand-children" builds.

MikeMcC399 added a commit to MikeMcC399/cy-docker-multistage that referenced this issue Dec 7, 2024
@ap-wtioit
Copy link

I'm also seeing a similar issue with https://github.com/Tecnativa/doodba where groupadd and useradd can only be done once

@MikeMcC399
Copy link
Author

MikeMcC399 commented Dec 11, 2024

I added a simpler reproduction to https://github.com/MikeMcC399/cy-docker-multistage which does not rely on cypress/factory:

git clone https://github.com/MikeMcC399/cy-docker-multistage
cd cy-docker-multistage
docker build . -f Dockerfile.parent -t parent --no-cache
docker build . -f Dockerfile.child-grandchild -t child-grandchild --no-cache

Dockerfile.parent

# syntax=docker/dockerfile:1
FROM debian:12-slim AS parent

RUN echo "parent"
ONBUILD RUN echo "parent to child"

Build Docker image parent:

$ docker build . -f Dockerfile.parent -t parent --no-cache
[+] Building 2.1s (9/9) FINISHED                                                                         docker:default
 => [internal] load build definition from Dockerfile.parent                                                        0.0s
 => => transferring dockerfile: 157B                                                                               0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                          1.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e7  0.0s
 => [internal] load build definition from Dockerfile.parent                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:12-slim                                                  0.5s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => CACHED [1/2] FROM docker.io/library/debian:12-slim@sha256:1537a6a1cbc4b4fd401da800ee9480207e7dc1f23560c21259f  0.0s
 => [2/2] RUN echo "parent"                                                                                        0.3s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:119f68760291ec6495317ed8e1b072b2bc6cd0f74931b70de40541c523185c14                       0.0s
 => => naming to docker.io/library/parent                                                                          0.0s

Dockerfile.child-grandchild

# syntax=docker/dockerfile:1
FROM parent AS child

RUN echo "child"

FROM child AS grandchild

RUN echo "grandchild"

Build Docker image child-grandchild from parent:

$ docker build . -f Dockerfile.child-grandchild -t child-grandchild --no-cache
[+] Building 2.7s (12/12) FINISHED                                                                       docker:default
 => [internal] load build definition from Dockerfile.child-grandchild                                              0.0s
 => => transferring dockerfile: 171B                                                                               0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                          0.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:db1ff77fb637a5955317c7a3a62540196396d565f3dd5742e7  0.0s
 => [internal] load build definition from Dockerfile.child-grandchild                                              0.0s
 => [internal] load metadata for docker.io/library/parent:latest                                                   0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [child 1/2] FROM docker.io/library/parent:latest                                                               0.0s
 => [child 2/3] ONBUILD RUN echo "parent to child"                                                                 0.3s
 => [child 3/3] RUN echo "child"                                                                                   0.5s
 => [grandchild 1/2] ONBUILD RUN echo "parent to child"                                                            0.6s
 => [grandchild 2/2] RUN echo "grandchild"                                                                         0.5s
 => exporting to image                                                                                             0.1s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:cfa55e2c974b1b397e904ff10a8bb0c7b60c9d00287f73d619680ed80bec24a1                       0.0s
 => => naming to docker.io/library/child-grandchild                                                                0.0s

The following log line from above should not appear. This ONBUILD RUN instruction should only be run in the child, not the grandchild:

 => [grandchild 1/2] ONBUILD RUN echo "parent to child"

@ap-wtioit
Copy link

ap-wtioit commented Dec 11, 2024

My current "simple" reproduction is:
Dockerfile.onbuild

FROM ubuntu AS onbuild
RUN touch /history.txt
ONBUILD ARG also_used_later=""
ONBUILD RUN echo "$(date) onbuild also_used_later: $also_used_later" >> /history.txt
ONBUILD RUN dd if=/dev/random count=$((1024*2)) > /random.bin
ONBUILD RUN sha256sum /random.bin >> /history.txt
ONBUILD ARG build_date
ONBUILD LABEL my_label=$build_date

Dockerfile.use_onbuild

ARG BASE_IMAGE
FROM $BASE_IMAGE AS first
RUN echo "$(date) first also_used_later: $also_used_later" >> /history.txt
FROM first AS second
RUN echo "$(date) second" >> /history.txt
FROM first AS clean
RUN rm /history.txt /random.bin
FROM scratch AS data
COPY --from=clean / /
COPY --from=first /history.txt /history-first.txt
COPY --from=second /history.txt /history-second.txt
ARG also_used_later
ENV also_used_later="$also_used_later"
RUN grep -RH '.' /history*.txt
# with clean build caches (docker system prune)
docker build -f Dockerfile.onbuild -t onbuild_image .
docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image .
docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image --build-arg "also_used_later=true" .

When runnning with DOCKER_BUILDKIT=1 BUILDKIT_PROGRESS=plain i get the following output from the grep (dumping the created history.txt) of the third image build (building a variant of the image in the second step):

#17 [data 4/4] RUN grep -RH '.' /history*.txt
#17 0.116 /history-first.txt:Wed Dec 11 14:41:26 UTC 2024 onbuild also_used_later: true
#17 0.116 /history-first.txt:1ffad5e071a575ea9a87654ea03c6e0c8a4f312d79c8eeb7738de8722aafa96e  /random.bin
#17 0.116 /history-first.txt:Wed Dec 11 14:41:27 UTC 2024 first also_used_later: true
#17 0.116 /history-second.txt:Wed Dec 11 14:41:26 UTC 2024 onbuild also_used_later: true
#17 0.116 /history-second.txt:1ffad5e071a575ea9a87654ea03c6e0c8a4f312d79c8eeb7738de8722aafa96e  /random.bin
#17 0.116 /history-second.txt:Wed Dec 11 14:41:27 UTC 2024 first also_used_later: true
#17 0.116 /history-second.txt:Wed Dec 11 14:41:27 UTC 2024 onbuild also_used_later: true
#17 0.116 /history-second.txt:a962687565ca6a6e2f1d7a948805c6dad30db692c293a1bd4cfcbcebb3680d26  /random.bin
#17 0.116 /history-second.txt:Wed Dec 11 14:41:27 UTC 2024 second

When running with DOCKER_BUILDKIT=0 the following result is achived

Step 14/14 : RUN grep -RH '.' /history*.txt
 ---> Running in d88e97868486
/history-first.txt:Wed Dec 11 14:41:08 UTC 2024 onbuild also_used_later: true
/history-first.txt:e142fb0793d6ea24f79f959f837ac30ca98ffb79157c203df08a65a5d0095b38  /random.bin
/history-first.txt:Wed Dec 11 14:41:09 UTC 2024 first also_used_later: true
/history-second.txt:Wed Dec 11 14:41:08 UTC 2024 onbuild also_used_later: true
/history-second.txt:e142fb0793d6ea24f79f959f837ac30ca98ffb79157c203df08a65a5d0095b38  /random.bin
/history-second.txt:Wed Dec 11 14:41:09 UTC 2024 first also_used_later: true
/history-second.txt:Wed Dec 11 14:41:09 UTC 2024 second

for us it does not depend on the dockerfile syntax, but we discovered it when updating our build servers one by one from:
dpkg -l | grep docker

ii  docker-buildx-plugin                   0.17.1-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
ii  docker-ce                              5:27.3.1-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                          5:27.3.1-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras              5:27.3.1-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose-plugin                  2.29.7-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  python3-docker                         5.0.3-1                                 all          Python 3 wrapper to access docker.io's control socket

to

ii  docker-buildx-plugin                   0.19.2-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
ii  docker-ce                              5:27.4.0-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                          5:27.4.0-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras              5:27.4.0-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose-plugin                  2.31.0-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  python3-docker                         5.0.3-1                                 all          Python 3 wrapper to access docker.io's control socket

So i'm not sure if it is the same issue or just a similar one.

Edit: also the following actions change the result to be equivalent in buildkit and non buildkit builds:

  • removing the ONBUILD ARG and ONBUILD LABEL for build_date
  • or skipping the docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image . build
    • but it then also happens if you run docker build -f Dockerfile.use_onbuild --build-arg BASE_IMAGE=onbuild_image --build-arg "also_used_later=true" . twice

@tonistiigi
Copy link
Member

@ap-wtioit This probably broke with #5490 . As a workaround you can pin to 1.10 as previous comment suggests.

@mzihlmann
Copy link

mzihlmann commented Dec 12, 2024

came here via git bisect 😄 but our issue is different

in our case the wrong base image is loaded. something alike

FROM python AS base
...
FROM base AS venv
...
FROM venv AS test
RUN some stuff
...

when we only build the test target, the build runs some stuff in the barenes python image, not the venv base target. Which then later fails because tools installed in the venv step are entirely missing. bisecting showed that this behaviour changed for us in 4cf7a1a too.

let me try and create a minimal reproduction project for you

@MikeMcC399
Copy link
Author

@mzihlmann
Copy link

Current PR so far doesn't fix the issue I'm running into, come to think of it it's probably quite distinct and I should open a separate issue thread for it. But there you go: https://gitlab.com/martizih/moby-syntax-issue

note how I invoke the build of the test target is a bit special:

docker buildx bake -f docker-bake.hcl test --set test.contexts.venv="docker-image://$IMAGE_VENV_TAG"

so instead of building our way there, we skip ahead by overriding the venv output with a previously built version. This is a bit pointless in this example I've given here, but does serve a purpose in our normal build environment, anyways it should work the same in both cases.

Thank you!

@MikeMcC399
Copy link
Author

@mzihlmann

It would be helpful to move your issue out of this issue thread. I don't see any use of ONBUILD in your example (which is the theme of this issue thread), and you say yourself that your issue is "probably quite distinct".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants