What files should be included in source and binary distributions?

[mcarans]

1. What are best practices for what should be included in Python source distributions and what gets included by default by setuptools?
2. What are best practices for what should be included in Python binary (wheel) distributions and what gets included by default by setuptools (assuming that what should be in a binary distribution differs from what should be in a source one)?

(eg. source files, configuration files, test files, test data, README.md, pyproject.toml, setup.cfg, LICENSE, requirements.txt, test-requirements.txt, documentation files).

Should this information be in setuptools documentation somewhere?

[abravalheri]

Hi @mcarans, I will try to give my personal opinion bellow, but first I will take the opportunity to make a (not so brief) introduction:

I like to imagine sdists and wheel as different optimisation stages between the code that you write in a VCS repository and the files that ended up installed in the user machine, more or less like the diagram bellow:

(A) VCS checkout ---> (B) sdist ---> (C) wheel ---> (D) package installed in the final location

In general the stages (A) and (B) are mostly identical (with the exception of metadata files), so for 90% of the cases sdists will be only another way of distributing your source code. However in some cases you might want to optimise a little bit more.

The classical examples of this sdist optimisation are packages using Cython: instead of adding the .pyx files to the sdist, some developers might want to pre-compile them to C code and add only the .c files. Another example would be people writing code in Hy or coconut that like to precompile their source code to Python files and include those in the sdist.

With that in mind, I would say that the main purpose of an sdist is to distribute packages in a platform/OS agnostic way. As long as that holds, it is up to the developers to decide if they want to publish exactly the code they wrote or if they want to pre-process/pre-compile it first.

On the other hand wheels are meant to be very platform dependent. In theory you can have a package installed by unzipping the wheel archive and simply moving the files to the right directories, no compilation required.

---

With this introduction out of the way, my personal approach is:

1. sdist - Add everything I would normally have in my repository to the sdist. I do like to include my test files and docs file together with a tox.ini (to orchestrate the tests and document generation). Including LICENSE, README, pyproject.toml, setup.cfg and all the files related to the project is pretty much a MUST. If you take this approach, I highly recommend setuptools-scm.

   (You can decide to leave out CI configuration files, text editor/IDE configuration files, or anything that is not strictly necessary to the package).

   By default setuptools will include all .py files in the package directories, LICENSE, README, .c files required by extensions and tests/test*.py.

2. wheel - Only the files that are required for your package to work. This files should be IMMUTABLE / READ ONLY and placed inside the package directory (this guarantees you can use importlib.resources or its backport to find these files during runtime).

   I don't recommend adding any file that can be configured by the users to the wheel, or data bases that can be written. Instead, if needed, I would write those files the first time my program launches to the user_data_dir or user_config_dir.

   By default setuptools will exclude everything outside the package directory from the wheel (e.g. docs, tests, examples). If you specify include_package_data = True any kind of file inside the package directory in the sdist is added to the wheel, otherwise setuptools will only include files that can be executed by Python.

[mcarans]

Very helpful, thanks!

[abravalheri]

and if I am not wrong .c files and the docs/, tests/ directories.

I was wrong, it is complicated what is added by default, but it should include at least these:

      - README, README.txt, README.rst, README.md
      - setup.py, setup.cfg
      - test/test*.py
      - all pure Python modules in the `py_modules` or `packages` configuration
      - all files pointed by the `package_data` configuration (inside package directories)
      - all C sources listed as part of extensions or C libraries
        in the setup script (doesn't catch C headers!)

from

setuptools/setuptools/_distutils/command/sdist.py

Line 210 in 1a1397a

def add_defaults(self):

(this list is extended a little bit in egg_info.py and sdist.py).

docs do not seem to be included.

While this default should be enough for most pure-Python packages, I like what setuptools-scm does.

[webknjaz]

I'd like to add that the downstream packagers (i.e. Fedora/CentOS/Ubuntu/Debian/Gentoo etc.) usually rely on the content of sdists when packaging their respective ecosystem formats like RPM or Deb. So it is important to include everything needed to build a project, or its docs, or run tests, into sdists — it should be possible to build wheels/RPMs/Debs by unpacking such a tarball and running commands without additionally accessing the project's Git repository.