Document how setuptools_scm works

[cdeil]

I've been reading through your docs https://github.com/pypa/setuptools_scm/blob/master/README.rst and I'm having a hard time to figure out how setuptools_scm works.

---

Question 1:

I see an option use_scm_version is passed to setuptools.setup.
Looking at the docs for setuptools https://setuptools.readthedocs.io/en/latest/ I can't find this option!? (might have missed it, unfortunately those docs don't have the normal Sphinx search field!?)
Is this option to setuptools.setup somehow an extension that's provided by the setuptools_scm package itself?

---

Question 2:

Does setuptools_scm ever generate a file with a version number?
The docs mention PKG-INFO under:
https://github.com/pypa/setuptools_scm/blob/master/README.rst#builtin-mechanisms-for-obtaining-version-numbers
Does setuptools_scm compute or directly store a version number in PKG-INFO or some other file?

---

Please consider this issue as a documentation request for people like me, that would like to understand how setuptools and setuptools_scm work and for which of my projects I should consider using setuptools_scm. (I guess all Python / git projects?)

Some extra sentences and links to relevant info (e.g. https://www.python.org/dev/peps/pep-0314/) would help.

Links to a few packages using setuptools_scm as "working examples" that I can look at and play around with before trying to modify my existing packages would also be very useful.

[RonnyPfannschmidt]

@cdeil thanks for bringing those points that need clarification to attention,
i will put a short rundown into this ticket so we can tackle a more detailed explanation for actual sphinx documentation later on

1. setuptools has an extension system to allow python packages to declare new keyword arguments to setup using entrypoints

this is similar to how console_scripts and gui_scripts works

2. PKG-INFO is a file generated by setuptools/distutils using the metadata from from the distribution object that is internal to setuptools/distutils

setuptools_scm has an optional tool that allows to write the version to text/python files using the albeit badly documented write_to argument

by default setuptools_scm does not write to any files, it only sets the version attribute of the distribution object used by setuptools/distutils

im using it with virtually all of my packges, - pytest-xdist, iniconfig, sentaku are just a few examples

[RonnyPfannschmidt]

another user was introduced by borgbackup/borg#161 :)

[WhyNotHugo]

Another point missing clarification is package data files.

It seems that setuptools-scm makes git-tracked file inside python modules get installed (if include_package_data is True), but this isn't clearly documented, and I've recently seen some breakage surrounding this (and I'm not even sure if something broke, or it only worked-by-coincidence before).

[RonnyPfannschmidt]

@hobarrera the surrounding breakage was setuptools ignoring sdist metadata in a major release, there is nothing we can do about that

[WhyNotHugo]

Right, I wasn't complaining about the breakage per se, just pointing out that the entire feature is mostly undocumented on setuptool-scm's side, so I wasn't sure if the previous behaviour was expected, or the current one was (eg: maybe I was just relying on a bug).

While we're on that topic though (slight OT for this thread though): this should be fixed in future by setuptools?

[RonnyPfannschmidt]

@hobarrera the setuptools bug will be fixed by setuptools,

btw, the file finders are mentionend in the readme, its not really comprehensible that way

unfortunately i rarely find the time to work on documentation atm

[maphew]

It took me some experimenting to figure out how to use the keywords instead of the default use_scm_version=True, so here's an example for the docs:

scm_version_options = {
    'write_to' : 'core/version.py'
    }

setup(
    ...
    use_scm_version = scm_version_options,
    setup_requires=['setuptools_scm'],
    ...
    )

[Kentzo]

@RonnyPfannschmidt Would you consider enabling wiki with community access?

[nealmcb]

Another thing I don't understand, along these lines. I'm using setuptools_scm just fine, somehow miraculously, since I never installed it. After searching a bit more, it seems that the setup_requires option does background installations of packages, not via pip, for any invocation of setup.py (even a "setup.py clean" step).
Ouch, as noted at what seems to me the aptly-named issue Avoid using setup_requires parameter in setup.py. #136 in another package.

Surely I'm missing something here. I'm used to installing the software I use to do a build up-front. That allows me to use the build software to find out what would be built, and, in this case, what version would be given to the package that is built. It also allows the from setuptools_scm import get_version command to work.

Can you document how to use setuptools_scm without requiring the use of setup_requires?
Or is it still necessary, even if this package is installed locally?

[RonnyPfannschmidt]

as soon as setuptools_scm is available in the local workingset it will no longer try to install it in .eggs

[RonnyPfannschmidt]

the general goal here is to slowly get people to use PEP-517/518 and let pip sort it out with a ephemeral env

[cdeil]

I tried setuptools_scm again today.

Couldn't figure out how to get a version number into the release sdist.

I have a git tag "v0.1" but apparently it's not found by setup.py. setuptools_scm only seems to work with pip, but to make a release, I need to run python setup.py sdist, no?

Looking through the docs of setuptools_scm I couldn't find this basic info concerning making a release.

😞

$ python setup.py build sdist
/Users/deil/software/anaconda3/envs/multinorm/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'use_scm_version'
  warnings.warn(msg)
/Users/deil/software/anaconda3/envs/multinorm/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'setup_requires'
  warnings.warn(msg)
/Users/deil/software/anaconda3/envs/multinorm/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
running build
running build_py
running sdist
running check
warning: check: missing required meta-data: version

reading manifest template 'MANIFEST.in'
writing manifest file 'MANIFEST'
creating multinorm-0.0.0
making hard links in multinorm-0.0.0...
hard linking LICENSE -> multinorm-0.0.0
hard linking README.rst -> multinorm-0.0.0
hard linking multinorm.py -> multinorm-0.0.0
hard linking setup.py -> multinorm-0.0.0
creating dist
Creating tar archive
removing 'multinorm-0.0.0' (and everything under it)

$ python setup.py egg_info
/Users/deil/software/anaconda3/envs/multinorm/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'use_scm_version'
  warnings.warn(msg)
/Users/deil/software/anaconda3/envs/multinorm/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'setup_requires'
  warnings.warn(msg)
/Users/deil/software/anaconda3/envs/multinorm/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'egg_info'

[RonnyPfannschmidt]

UUse setuptools ...

[RonnyPfannschmidt]

#343 (comment) was merged as a starting point, we still need more

[embray]

I was definitely confused at first as well about exactly when write_to gets invoked, especially since I put [tools.setuptools_scm] in my pyproject.tml (with a write_to option). But it was only when I added it to my setup.py as well that write_to was being invoked when I run pip wheel to build a wheel for my package (then it works fine and exactly as I had hoped).

Another bit missing from the docs is recommendations for what to do when using a package that's installed in editable mode (pip install -e .), or even just being imported from the pwd. My colleagues would like pkgname.__version__ to always show the current local version without having to re-install the package (which is otherwise the only way the package metadata gets updated).

Currently I came up with something like:

try:
    from ._version import version as __version__
except ImportError:
    # Possibly running in the git repository
    try:
        import setuptools_scm
        __version__ = setuptools_scm.get_version()
        del setuptools_scm
    except Exception:
        warnings.warn(f'could not determine {__name__} package version')
        __version__ = '0.0.0'

This seems to work, and under a "normal", non-development installation the _version.py file exists, so setuptools_scm never needs to be invoked at runtime. Only during development, which is acceptable.

In any case thanks @RonnyPfannschmidt for continuing to maintain this package! I haven't used it in many many years and it's evolved tremendously since I last did. Even despite these confusions it implements I think exactly what my colleagues are clamoring for and saved me a ton of time in not having to roll my own.

[RonnyPfannschmidt]

@embray thans for those encouraging words,

currently there is no concept of editable install aware versioning, as setuptools doesn't provide the tooling to actually update metadata (aka the package metadata will be incorrect until the next refresh)

i wonder if perhaps a install-able post-commit/checkout hook could be used to update the package metadata on git usage in a reliable manner

[embray]

i wonder if perhaps a install-able post-commit/checkout hook could be used to update the package metadata on git usage in a reliable manner

I'm not sure that would fully solve the problem; for example it wouldn't detect when the repository is dirty.

I think solutions similar to the one I posted above, where setuptools_scm.get_version() is just called at runtime (but only if run from the git repo, for example) solve the problem well-enough.

There are some corner cases where this is "dangerous", like if some source code gets copied into a different git repo (maybe I extract a source tarball into some miscellaneous repo and then install it, for example). This is mitigated somewhat by first looking for the version.py module, which should be included in the source tarball.

I've also mitigated this kind of issue before by looking for a special canary file that would only possibly exist in my project's repo. For example:

with open(os.path.join(os.path.dirname(__file__), '..', '.canary')) as fobj:
    in_my_project_repo = fobj.read() == '<some hard-coded randomly generated string>'

Maybe overkill in most cases, but I've found this pattern useful in the past when I want to be really sure that my code is running out of the correct repository.

[RonnyPfannschmidt]

@embray i like that idea
@jaraco say - is there a way to determine if the current distribution thats "to be installed" will be part of a install/bdist_wheel or part of a editable install

[jaraco]

@jaraco say - is there a way to determine if the current distribution thats "to be installed" will be part of a install/bdist_wheel or part of a editable install

I don't think so. Commands are constructed with the distribution object, so the dependency works in the other direction. Also, a distribution is theoretically a more abstract concept -- it's meant to exist independent of any command that executes on it.

[GDYendell]

I am also trying to implement a way to get the runtime version if editable and the install version otherwise. I initially thought we could try to read from the generated _version.py and if it doesn't exist use setuptools_scm.get_version, e.g.

try:
    from _version import version
    __version__ = version
except ImportError:
    # We are not installed - get runtime version
    from setuptools_scm import get_version
    __version__ = get_version()

but this file is also created in editable installs.

Is it viable for the write_to file not to be created for editable installs, perhaps as an option? I think this would be quite a neat solution if so. Otherwise I expect we will also go for the check if magic non-installed file exists method. Or try setuptools_scm.get_version first and if that fails because it is not the root of the git repo, use the install version

[RonnyPfannschmidt]

@GDYendell please open a new issue for this topic

Currently there is no reasonable way to tell the difference between editable installs and normal install

Im inclined to rather supplement a git hook that updates the files /package metadatas rather than a error prone hackish file with magical version extraction

[RonnyPfannschmidt]

closing this as complete with the merge of #880 adding more documentation