provide docker image(s)

[casperdcl]

Provide docker images

• make dev installation and testing easier
• also can be used by end-users (something like docker run --rm -t -v $PWD:/repo dvc status)
• also would make it easier for end-users to incorporate into their own docker images
• can be tested on travis
• can be released via github package registry
• can be released via docker hub
• related Snapcraft #2771
• related consider getting rid of packages/binaries built by pyinstaller (windows exe and mac pkg) #2760

[ghost]

@casperdcl , I'm not sure how much benefit would bring, giving that it is not that hard to create a docker image for DVC: docker run --name dvc -ti python pip install dvc && docker container commit --change 'CMD ["dvc"]' dvc dvc

Also, testing this would be the same as testing DVC on a linux machine (or at least, it should be)

I doubt that users want to use it as a base image, since it doesn't provide anything else than DVC.

I'd prefer to not maintain this one, to be honest

[pared]

I would also argue, that when doing some data science stuff in docker, its probably easier to install DVC in your own image, rather than try to adjust DVC image to you requirements (like installing TF/pytorch/...).

[casperdcl]

@pared that was my point about

• also would make it easier for end-users to incorporate into their own docker images

as in people could copy-paste from our docker file into theirs...

I assumed it's more complex than just a pip install to get all the features (at least some apt-get installs as well?)

[pared]

@casperdcl ok, I didn't quite get it.

as in people could copy-paste from our docker file into theirs...

That would surely help to build custom image.

[efiop]

@casperdcl Hm, apt-gets would depend on the base image that you are using. If it is absolutely bare then yeah, you will need to install python and whatnot. But for regular things like ubuntu you don't have to install anything special, except, I guess, git. We have a docker image that we are using for testing here https://github.com/iterative/dvc-test/blob/master/docker/ubuntu/16.04/Dockerfile . Not sure if we still need to install libffi explicitly 🤔

[casperdcl]

@efiop the setup.py file seems to imply more deps are required - I was referring to a full pip install dvc[all,ssh_gssapi,tests]

[casperdcl]

@efiop can confirm libffi not required on ubuntu:18.04

[efiop]

@casperdcl Ah, got it. yeah, gssapi requires some dev tools to compile stuff as well as tests, so the dockerfile for that would be more complicated. We could provide it in the docs maybe? I'm not quite sure about building and testing it. When used for regular tests, it would always run linux inside, instead of running on the host system, which is a problem, as we already have a lack of native-testing on windows and osx, and that might prevent us from discovering bugs when developing. Though a nice thing about it is that you don't have to setup a test env on your machine, that is true 🙂

[casperdcl]

Yes I did start making a few flavours of docker images for testing (alpine, ubuntu LTS, 2.7, 3.6 etc) ages ago which are probably sitting in a git stash on one of any number of machines. maybe.

Now I just use a conda env for dvc testing.

[casperdcl]

btw totally fine with this issue being closed - don't actually have any strong opinions about it.

[efiop]

@casperdcl No reason to close it. Docker images(or at least dockerfiles) would be nice to have, for sure. 🙂

[nettoyoussef]

Provide docker images

I just came here looking exactly for that.
Every time I see a comment like:
"It's highly recommended using virtual environment or pipx (on Python 3.6+) to encapsulate your local environment."

I instantly go looking for a docker file for me to easily test the software.

That is because I use different software (mostly R), and don't use python outside docker (because python environment libraries change a lot, and nobody seems to agree which one is best - conda, virtualenv, pyenv, pipenv etc - which, to complicate further, have different functionality).

[efiop]

@nettoyoussef Thanks for the feedback! Do you need pre-built images, or a Dockerfile in our docs would do?

[dmpetrov]

@casperdcl thank you for this idea!

I agree with @pared and @MrOutis that it is easy to create your own docker image and it might create additional supporting overhead for us.

However, prebuild docker gives value to users and @nettoyoussef showed some example. It can attract users' attention and improve usability despite a simple implementation.

But then documentation plays the major role. Can we make a good documentation page or even a small blog post that explains the motivation behind using docker image instead of installed tool and when it is needed? Why don't we start with doc/blog-post and then implement the docker image.

[nettoyoussef]

@nettoyoussef Thanks for the feedback! Do you need pre-built images, or a Dockerfile in our docs would do?

Thank you for being so helpful.

Personally, a Dockerfile would suffice. The community, however, maybe would benefit more from a pre-built image.

Instead of building one from Ubuntu, you could make your life easier and, e.g., start from a miniconda image. I think this can be easy to automate, and maybe you can even delegate this to other teams - a partnership with rocker for example.

Since from what I read DVC is not attached to any particular library/language it also makes sense to separate concerns, i.e., you can use the same DVC image with any project, instead of installing it with any particular environment.

It also makes easier to implement it in existing projects - since you don't have to rebuild the images just to try it.

[casperdcl]

maybe I'm missing something but it looks like they're using index.js (node) rather than the Dockerfile (docker). Seems like the GH Action should really use the docker image (e.g. https://github.com/casperdcl/covid-19-box).

On a related note I like where this is heading https://github.com/iterative/dvc-cml/wiki/Tensorflow-Mnist-for-Github-Actions

[dmpetrov]

@casperdcl it is using docker files. Index.js is here just support GH users who don’t want or cannot use docker. You can find it in the workflow files.

[casperdcl]

Right. Seems a bit odd to provide a nodejs action for public use via the standard uses: syntax, but in our workflow use our own docker version. Unrelated to this issue (#2774) though.

[dmpetrov]

@casperdcl what would be your suggestions for that project? How to organize it in the right way?

[casperdcl]

1. In action.yml, use Dockerfile
2. Move as much of the other non-essential root clutter to subdirs
3. The Dockerfile's entrypoint itself can use node and/or any other software that users demand support for
4. In our workflow,
   • use node/black/flake8/etc directly to run linting/tests (as currently)
   • uses: ./ to run/test the (docker) action

Advantages:

• a single CI run of our provided action installs and uses/tests everything (docker wraps node, etc.)
• local installs and our workflow tests can continue to manually install prerequisites and use just node (as currently)
• both cloud (other people's workflows as well as our own) and local installs can use the convenient docker wrapper/action (with all of its advantages such as dependency management & reproducability)

Surely should discuss this in an issue on that repo though?

[shcheklein]

@casperdcl

Move as much of the other non-essential root clutter to subdirs

you mean prettier configs, etc, etc?

The Dockerfile's entrypoint itself can use node and/or any other software that users demand support for

could you elaborate?

uses: ./ to run/test the (docker) action

same here, could you elaborate?

[casperdcl]

you mean prettier configs, etc, etc?

er, just a general principle removing as much as possible. Some tools expect files to be in the root so we're mostly stuck there, ofc.

you mean prettier configs, etc, etc?

It's cumbersome for us to maintain multiple, well, entrypoints to our actual code. If we want to support both docker and directly running in node, it's best to have docker be a thin wrapper around node (i.e. in the Dockerfile, use ENTRYPOINT npm, CMD run or similar).

This way we can use the docker wrapper for the action. Thus running the action will test our docker wrapper as well as the underlying entrypoint. The additional advantage is that all deps are guaranteed installed in the docker container.

[shcheklein]

I feel that I'm still missing something :)

Docker entrypoint for the image we provide already does this, right? It already runs Node. And image itself has JS bundle pre-installed.

There are no very strong reason to support direct docker-less action, but it's a separate topic.

[casperdcl]

yes I was making several minor points, I think we're all missing small things but nothing major :)

[DavidGOrtega]

Hi! we are pushing to use the code through Docker, the main reasons are:

• Complexity abstraction (The container acts as a wrapper of the whole stack)
• User experience is the same (or very close) in Gitlab and Github
• Stack is preinstalled so its faster to execute
• Less friction and ready to go for GPU
• The image will also act as a self runner, again, with the same user experience in Gitlab and Github.

The MAIN reason why the js action is maintained is because there is no way MACOS or WINDOWS can run specific native tools in docker. So if a user would be using i.e. CoreML with Xcode the only way to make this work available for them is through the purely Github Js action and only in Github

[hsharrison]

Hi, we're using the dvcorg/cml image, but it's creating a pain point that there are no version tags, only latest. It would be great if images were tagged by version so we could pin it and enforce the same version as in local dev environments.

[skshetry]

Hi, @hsharrison. Could you please mention your issue in the CML's repo, specifically on iterative/cml#217? This way, it'll be easier and faster for you. Thanks.

[DavidGOrtega]

👋 @hsharrison feel free to open a ticket there. Could you please provide also whats your pain point?
We decided to not tag it (despite that we have already a ticket for that created by me) since we always try to do CML backwards compatible.

[hsharrison]

My mistake, sorry for the noise.

[kinow]

Hello,

I couldn't find instructions for containers in the installation page of DVC, and I found this discussion after searching for docker in the issues list. Reading the comments above, I think there was no clear rationale for adding a container, even though there were at least two attempts:

• [WIP] install: add docker usage documentation dvc.org#811
• docker: add example Dockerfile #2844

I think I understand the resistance to add a Dockerfile or instructions without a clear reason for doing so, especially as it would be something else to have to maintain along with the code, tests, docs, pip/snap/etc installation methods. But I thought if users provide the reason why they thought about this, maybe that could help a little.

In my case, I work with HPC climate workflows. These workflows run climate models with large datasets, and we have recently started tests using DVC instead of managing model data the traditional way (hard-coded paths on HPCs). This is looking promising, especially as we are using multiple HPCs over Europe.

One issue with HPCs, is that many environments are either offline/secluded, or may have older versions of Python, or have issues resolving dependencies. It's really very likely that you won't be able to get the same version of a tool installed with pip on 2 HPCs. In our case we have at least 4, plus internal development environments.

Singularity containers are used where possible, to address this issue of portability, and installing tools like DVC in the HPC. I am quickly creating a container with python or micromamba where I will just do a pip install dvc==$version, but if a project provides documentation for containers, I find it useful as it saves time in case there are any gotchas when creating the containers for these tools.

So it would be useful to have, if not a Dockerfile or container, then just a brief explanation in the docs saying that it is recommended to use pip or pipx in a container, and that there are no environment variables or volumes/files required to use DVC, or if there is anything that the user must be aware of when using DVC within containers.

Thanks for DVC!

-Bruno