Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WiP] Draft for Gentoo configuration #611

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
27ca573
Draft for Gentoo configuration
TheChymera Apr 12, 2024
304d24e
Use HTTPS
TheChymera Apr 12, 2024
32cd210
Adding test template to develop gentoo support
TheChymera Apr 25, 2024
91df306
Added containerfile contents
TheChymera Apr 25, 2024
3428ab9
Corrected command
TheChymera Apr 25, 2024
7f72cf7
Trying to add package manager support
TheChymera Apr 25, 2024
01fd95a
ignore Yarik's venvs/
yarikoptic Apr 30, 2024
a5df85f
We are not yet fully ready for gentoo in recipes
yarikoptic Apr 30, 2024
39b79fc
Provide base skeleton for gentoo to pass test_cli tests
yarikoptic Apr 30, 2024
6bca6f5
Add some demo options for instructions -- the problem is that it is f…
yarikoptic Apr 30, 2024
9b0d75f
image versions not handled in this template, reusing parameters for hash
TheChymera May 1, 2024
c8cb922
Embedding Gentoo image resource identifiers into the logic
TheChymera May 1, 2024
456c479
Making gentoo template/invocation work (more TODOs)
yarikoptic May 24, 2024
30ab193
Added new hashes
TheChymera May 28, 2024
e7f1720
Updated hashes and dates after test, provide AFNI with version fix as…
TheChymera May 29, 2024
758951b
updated hashes
TheChymera Jun 7, 2024
8b5df1d
Rudimentary fix to blunt stab at a test + added TODOs
yarikoptic Jun 11, 2024
722e555
Removed config tracking in ancillary files
TheChymera Jul 29, 2024
5420c04
Updated hashes
TheChymera Jul 30, 2024
54c98f6
better package for testing
TheChymera Jul 30, 2024
5fd6303
Even smaller package for testing
TheChymera Jul 30, 2024
cc0d86e
flake8 fixes
TheChymera Aug 2, 2024
6887052
Merge branch 'master' into jen2
Remi-Gau Aug 15, 2024
6d76eb5
run precommit
Remi-Gau Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions neurodocker/cli/tests/test_build_images_with_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,23 @@ def test_json_roundtrip(cmd: str, inputs: str, tmp_path: Path):
stdout, _ = run_fn(img, args=["env"])
assert "CAT=FOO" in stdout
assert "DOG=BAR" in stdout

def test_gentoo_image(tmp_path: Path):
# also add singularity like in the test above

cmd = "docker"

_TemplateRegistry._reset()
runner = CliRunner()
result = runner.invoke(
generate,
[
cmd,
"--pkg-manager apt",
"--base-image neurodebian:bullseye",
"--ants version=2.4.3",
"--user nonroot"
],
)
assert result.exit_code == 0, result.output
(tmp_path / "specs.json").write_text(result.output)
4 changes: 2 additions & 2 deletions neurodocker/reproenv/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
allowed_installation_methods = {"binaries", "source"}
installation_methods_type = Literal["binaries", "source"]

allowed_pkg_managers = {"apt", "yum"}
pkg_managers_type = Literal["apt", "yum"]
allowed_pkg_managers = {"apt", "portage", "yum"}
pkg_managers_type = Literal["apt", "portage", "yum"]

# Cross-reference the dictionary types below with the JSON schemas.

Expand Down
2 changes: 2 additions & 0 deletions neurodocker/templates/afni.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ binaries:
- which
- unzip
- ncurses-compat-libs
portage:
- sci-biology/ants
debs:
- http://mirrors.kernel.org/debian/pool/main/libx/libxp/libxp6_1.0.2-2_amd64.deb
- http://snapshot.debian.org/archive/debian-security/20160113T213056Z/pool/updates/main/libp/libpng/libpng12-0_1.2.49-1%2Bdeb7u2_amd64.deb
Expand Down
49 changes: 49 additions & 0 deletions neurodocker/templates/gentoo.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
# Instructions to add NeuroDebian repositories.

name: gentoo
url: https://www.gentoo.org/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is nothing gentoo below yet, misleading commit message etc. Please squash it with whatever actually gentoo specific changes you place into this PR

binaries:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
binaries:
binaries:
arguments:
required:
- portage_version
- stage3_version
optional:
- gentoo_hash
- science_hash

TODO: defaults somehow -- may be just straight in the code as {{ self.get("portage_version", "20240324") }}

instructions: |
FROM docker.io/gentoo/portage:20240324 as portage
FROM docker.io/gentoo/stage3:20240318
# copy the entire portage volume in
COPY --from=portage /var/db/repos/gentoo /var/db/repos/gentoo
RUN emerge -v --noreplace dev-vcs/git
RUN emerge -v1u portage
# Pinned commits for the dependency tree state
ARG gentoo_hash=2d25617a1d085316761b06c17a93ec972f172fc6
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARG science_hash=73916dd3680ffd92e5bd3d32b262e5d78c86a448
ARG FEATURES="-ipc-sandbox -network-sandbox -pid-sandbox"
# This will be bound, and contents available outside of container
RUN mkdir /outputs
COPY .gentoo/portage/ /etc/portage/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: copy from https://github.com/TheChymera/neurodocker_g/tree/master

but in the future some might also want to become templates

# Moving gentoo repo from default rsync to git
RUN rm /var/db/repos/gentoo -rf
# Cloning manually to prevent vdb update, pinning state via git
# Allegedly it's better to chain everything in one command, something with container layers 🤔
RUN \
REPO_URL=$(grep "^sync-uri" /etc/portage/repos.conf/gentoo | sed -e "s/sync-uri *= *//g") && \
mkdir -p /var/db/repos/gentoo && pushd /var/db/repos/gentoo && git init . && \
git remote add origin ${REPO_URL} && \
git fetch --filter="blob:none" origin $gentoo_hash && \
git reset --hard $gentoo_hash && rm .git -rf && popd && \
REPO_URL=$(grep "^sync-uri" /etc/portage/repos.conf/science | sed -e "s/sync-uri *= *//g") && \
mkdir -p /var/db/repos/science && pushd /var/db/repos/science && git init . && \
git remote add origin ${REPO_URL} && \
git fetch --filter="blob:none" origin $science_hash && \
git reset --hard $science_hash && rm .git -rf && popd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all the "freezing" might make sense to move into a script. this would allow for better reuse...

also note that apparently for neurodebian -- freezing is done somewhere else than this template

# Old Christian: Remove sync-uri to not accidentally re-sync if we work with the package management interactively
# Christian from the future: Maybe we want the option to re-sync if we're debugging it interactively...
#RUN sed -i /etc/portage/repos.conf/{gentoo,science} -e "s/sync-type *= *git/sync-type =/g"
#RUN sed -i /etc/portage/repos.conf/{gentoo,science} -e "/sync-uri/d"
#RUN sed -i /etc/portage/repos.conf/{gentoo,science} -e "/sync-git-verify-commit-signature/d"
# Make sure all CPU flags supported by the hardware are whitelisted
# This only affects packages with handwritten assembly language optimizations, e.g. ffmpeg.
# Removing it is safe, software will just not take full advantage of processor capabilities.
#RUN emerge cpuid2cpuflags
#RUN echo "*/* $(cpuid2cpuflags)" > /etc/portage/package.use/00cpu-flags
### Emerge cool stuff here
### Autounmask-continue enables all features on dependencies which the top level packages require
### By default this needs user confirmation which would interrupt the build.
RUN emerge --autounmask-continue afni fsl
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
RUN emerge --autounmask-continue afni fsl
# RUN emerge --autounmask-continue afni fsl

Loading