fastconda

Get packages onto your conda channel faster

Status

Builds

Package Name	Pypi	GitHub	Conda fastai channel
sentencepiece
opencv-python-headless		N/A
timm
albumentations
imgaug

Copies

Package	Pypi	Conda fastchan channel
cudf
cudatoolkit	N/A
mamba
pytorch
torchvision
transformers
rich
sentencepiece
fastai
timm
nbdev
fastrelease
ghapi
fastcgi

Build Process

We are using three different ways for sourcing a Conda package into an Anaconda repo:

1. conda build: When there are C dependencies.
2. setuptools-conda: For pure python packages.
3. anaconda copy: When a maintained Anaconda package already exists.

conda build

When there are C dependencies.

Build a Conda package by first installing the appropriate pip package(s) in a fresh Conda environment, and then use conda build to build a package based on this environment. We do this for packages that have C dependencies and need thus need binaries created for different platforms. This build process is specified in condabuild.yml. This type of build requires a specific directory structure with several metadata files which amounts to a fair amount of boilerplate. For this reason, we dynamically generate all of this boilerplate based on the configuration file build.yaml

The schema of build.yaml is as follows:

- pypinm: opencv-python-headless
  import_nm: cv2 # Optional: if not specified will default to the same as pypinm
  deps: [numpy]  # Optional: if not specified will just be python
  path: destination_dir # Optional: if not specified, files will be placed in a directory named after `pypinm`
- pypinm: sentencepiece # This second package, `sentencepiece`, uses all of the defaults.

In the above example, we specify two packages to built, opencv-python-headless and sentencepiece. Here is a description of all fields:

• pypinm: this field is required and is the name of the package on pypi.
• import_nm: You only need to supply this field when the import name of the pypi package is different than the package name. For example the import name of opencv-python-headless is cv2.
• deps: This is a list of all dependencies. If this is not supplied, no dependencies beyond python are assumed.
• path: You should usually ignore this field completely and rely on the default behavior. This optional parameter allows you to specify the directory where the metadata files will be placed. If not specified, files will be placed in a directory named after pypinm.

You can run this locally with:

python build.py

see condabuild.yml for necessary environment setup.

setuptools-conda

For pure python packages.

For python packages that are pure-python that do not require binaries, we can instead create a cross-platform Conda package using setuptools-conda. This build process is specified in setupconda.yaml.

You can run this locally with:

./setupconda.sh {args}

see setupconda.yaml for example of args

anaconda copy

When a maintained Anaconda package already exists.

In situations where there is a reliable and maintained conda package already present in another channel, we can copy this package and all its dependencies to another channel. This is desirable when you want to simplify and speed up the installation of packages by placing all dependencies in a single channel. This process is carried out via anacopy.yml. We find all dependencies for a particular package by doing a Conda installation, which uses the Conda solver to find all the dependencies with appropriate version numbers, and then copy the appropriate packages using anaconda copy.

You can run this locally:

python get_deps.py

See anacopy.yml for the full workflow.