[BUG] "'extras_require' must be a dictionary..." regression

[rt121212121]

setuptools version

setuptools==67.0.0

Python version

Python 3.10

OS

Linux / Ubuntu

Additional environment information

No response

Description

With latest setuptools I am unable to use a package (btchip-python) as input to pip-compile requirements for hash generation. It errors with:

error in btchip-python setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

If I downgrade setuptools to a version I know works, 65.5.0, the package works again.

The problem part of the project's setup.py file is:

    extras_require = {
	'smartcard': [ 'python-pyscard>=1.6.12-4build1' ]
    },

It is triggered in the following command pip-compile does:

python3 setup.py egg_info

Expected behavior

Unless there is an obvious reason that setuptools is rejecting values it accepted before, I expect it to continue to accept them and no do breaking changes.

How to Reproduce

The simplest reproduction case is taking the erroring command from pip-compile and running it in a local copy of the package.

1. pip3 install setuptools==67.0.0
2. git clone https://github.com/LedgerHQ/btchip-python.git
3. cd btchip-python
4. python3 setup.py egg_info
5. Observe that it errors with the extras_require complaint.

Then with the older setuptools.

1. pip3 install setuptools==65.5.0
2. python3 setup.py egg_info
3. Observe that it generates the eggy stuff.

Output

vboxuser@Ubuntu2204:~/_src/btchip-python$ python3 setup.py egg_info

error in btchip-python setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

vboxuser@Ubuntu2204:~/_src/btchip-python$ pip3 show setuptools

Name: setuptools

Version: 67.0.0

Summary: Easily download, build, install, upgrade, and uninstall Python packages

Home-page: https://github.com/pypa/setuptools

Author: Python Packaging Authority

Author-email: [email protected]

License: 

Location: /home/vboxuser/.local/lib/python3.10/site-packages

Requires: 

Required-by: pip-tools

vboxuser@Ubuntu2204:~/_src/btchip-python$ pip3 install setuptools==65.5.0

Defaulting to user installation because normal site-packages is not writeable

Collecting setuptools==65.5.0

  Downloading setuptools-65.5.0-py3-none-any.whl (1.2 MB)

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.4 MB/s eta 0:00:00

Installing collected packages: setuptools

  Attempting uninstall: setuptools

    Found existing installation: setuptools 67.0.0

    Uninstalling setuptools-67.0.0:

      Successfully uninstalled setuptools-67.0.0

Successfully installed setuptools-65.5.0

vboxuser@Ubuntu2204:~/_src/btchip-python$ which pip3

/home/vboxuser/.local/bin/pip3

vboxuser@Ubuntu2204:~/_src/btchip-python$ python3 setup.py egg_info

running egg_info

creating btchip_python.egg-info

writing btchip_python.egg-info/PKG-INFO

writing dependency_links to btchip_python.egg-info/dependency_links.txt

writing requirements to btchip_python.egg-info/requires.txt

writing top-level names to btchip_python.egg-info/top_level.txt

writing manifest file 'btchip_python.egg-info/SOURCES.txt'

reading manifest file 'btchip_python.egg-info/SOURCES.txt'

reading manifest template 'MANIFEST.in'

adding license file 'LICENSE'

writing manifest file 'btchip_python.egg-info/SOURCES.txt'

vboxuser@Ubuntu2204:~/_src/btchip-python$

[abravalheri]

Hi @rt121212121 ,

The requirement python-pyscard>=1.6.12-4build1 has an invalid version (ref: PEP 440, PEP 508).

[pelson]

I see this problem even with:

>>> from packaging.requirements import Requirement
>>> Requirement('requests=>2.8.*')

From what I can see, this is unexpected as per https://peps.python.org/pep-0508/#extras. This will be a problem with the packaging library, if it is indeed a problem.

UPDATE: I have found the underlying issue pypa/packaging#645, which is causing this for my case (though not the case for the original question). In my case, it is by design (even if it is confusing/unexpected).

[abravalheri]

Hi @pelson, yes, I believe that is a separated problem and it is related to one of setuptools dependencies (packaging).

[rt121212121]

Hi @rt121212121 ,

The requirement python-pyscard>=1.6.12-4build1 has an invalid version (ref: PEP 440, PEP 508).

Yes, this is a regression (apparently in the packaging library). Either the breakage we are now seeing is ordained as the right path forward and packages that used to work, will no longer work and will have to be updated. A packages I do not maintain that I have been using for a decade is affected by this :-) Or we restore those packages to working and fix the regression.

[abravalheri]

Hi @rt121212121, I am afraid this is not a regression but an intentional change that has been planned in the ecosystem for a while. You can have a look on this comment for a reference:

• InvalidVersion exception when invalid version used on Setuptools 66 #3772 (comment).

From issue tracking management point of view, I believe that the best is to close this issue for now since this is a direct consequence of issue 3772 (also due to the fact that the changes would have to come from packaging first).

[rt121212121]

At the end of the day, you have to do what is right for you.

But please keep in mind that as a Python developer I am continually finding things are breaking and then spending a day looking into it, then filing bugs and being told it is "by design". This happens for the Python 3.11 release where there are unadvertised breaking changes that look like bugs, and it happens for other things including Setuptools now.

These decisions cost me as a developer, they cost my employer, and they make it much much more unlikely to ever bother submitting bugs as the bugs are indistinguishable from features ... by design.

[abravalheri]

Thank you very much for the comprehension @rt121212121.

To help with this problem in the future (avoid loosing time with breaking changes) you can try running your test suite with all warnings turned into errors by default ¹ and selectively allowing minor warnings as needed² (or at least with Python development mode activated). This should help you identify upcoming breakages and be prepared before they happen.

In setuptools we have been trying to communicate changes before hand and minimize the cost for the users as much as we can (and as much as it is viable for the project), using warnings and the changelog. Sometimes we will fail and I apologize for that, but Setuptools is an old code base and we have limited amount of time to work on it.

Footnotes

1. You can use Python's -W option or pytest config for that. ↩

2. It is unfortunate that Python silence deprecation errors by default. ↩

[jeff-hykin]

For those who just need to get their project working pip install setuptools==57.1.0 then retrying my orignial command (pip install stable-baselines3) then worked for me.

[timkpaine]

@rt121212121 if you hit this in your own library and you're defining extras_require in a setup.py, try adding the following to your pyproject.toml:

[project.optional-dependencies]

Looks like by default, trying to get this field when it is missing results in a return type of [], whereas when you have it present (even if empty) its results in {}.

(@abravalheri so there might actually be a bug here, the default value of trying to fetch project.optional-dependencies when it is missing from pyproject.toml should be a dict not a list, e.g.

setuptools/setuptools/config/_apply_pyprojecttoml.py

Line 412 in 2c28939

"optional-dependencies": [],

)