TestPyPI: "ERROR: No matching distribution found for setuptools" for PEP517-style packaging (with pyproject.toml)

[normanius]

I would like to forward this report of a problem related to the TestPyPI infrastructure: pypa/pip#9242

PEP517-style package installation via the TestPyPI infrastructure fails because (the wording might be wrong) TestPyPI does not serve setuptools that is required to build the package under isolation.

TestPyPI is advertised by popular tutorials used by packaging newbs. So I guess it'd be good if this is fixed or at least the beginners get some guidance on this issue.

Note that the same package installed through the real PyPI does not show this problem.

@uranusjr suggested a workaround:

If you need to test the installation right now, the easiest fix available without involving others would be to serve your own setuptools copy instead. Something like

pip wheel setuptools wheel -w ./wheels
pip install --find-links ./wheels --index-url https://test.pypi.org/simple ...

[jaraco]

Thanks for the report. As pointed out in the upstream pip ticket, fixing this issue for Setuptools, which would require an ongoing commitment to cut releases both to TestPyPI and PyPI, would still only be a stopgap for this one symptom. The broader symptom is that users can't expect all or even many of projects published to PyPI to be published to TestPyPI.

I second the recommendation to change the workflow, either to use --extra-index-url or to use another test index like devpi instead of TestPyPI as the former offers PyPI mirroring support (and better security around mirrored packages).

Setuptools would be happy to participate in an effort to populate popular packages into TestPyPI, but until there's an organized effort to do that across the ecosystem, I'd rather not invest the energy in doing it.

[normanius]

I see. I just hit the tip of an iceberg. Do you recommend using following command?

pip install --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple/ \
            --no-deps \
            some-package-name

Just to make sure I correctly understand: Does this try to fetch the packages first from TestPyPI, and if the stuff is not available, will resort to PyPI?

I think this information should be spread, as it is currently one of the hurdles that cause beginners to stumble when adopting PEP517-style packaging. I have this popular tutorial in mind. At the moment, there is no reference to such issues.

[jaraco]

After recommending --extra-index-url and in consideration of your question, I did some more investigation and found that the behavior of extra-index-url is non-deterministic, so I don't think that's a good solution either. Their solution was to build a mirror (PyDist, sounds similar to devpi).

I think the broader issue (challenges developing under PEP 517 with TestPyPI) needs to be raised in a broader forum, probably discourse. Sorry to bounce you around, but you've definitely exposed some unexpected challenges.

[uranusjr]

--extra-index-url is deterministic. The issue is it’s not what people think it is 🙂

It is viable to use --extra-index-url for this, but not pointing it to pypi.org.

[normanius]

@uranusjr I don't understand. Why isn't it viable to point it to pypi.org?

[uranusjr]

It is not a good idea to use --extra-index-url=https://pypi.org/simple here because the search results of --index-url and --extra-index-url are merged, not first-found-first-win. So if pypi.org has a package 2.1 while test.pypi.org has only package 2.0, package 2.1 from PyPI will be chosen over package 2.0 on TestPyPI. This is usually not the behaviour you’re looking for when using TestPyPI.

[normanius]

Thanks for the clarification.

As suggested, I started a discussion on discourse. Hope this helps.

[vernondcole]

Since this issue is still open, and since my issue (though from an extremely different use case) has the same error message, I think that it might be better to comment here than to open a new issue.

My use case is installing source packages on machines in a secure environment -- that is, where the target computers have no Internet access, ever. After converting a simple package to use pyproject.toml rather than setup.py, and testing and finding all well on a development environment, I made my first attempt at installing on an isolated machine. After many hours of frustration, I finally found a link with the correct magic. "use --no-build-isolation" feature.

I am sure that someone feels that there is some really good reason for downloading a fresh copy of setuptools across the Internet rather than use the perfectly good one which is already on my local disk. Perhaps so. As a guy who was really happy to get his first 1200 baud modem, it seems pretty wasteful. Maybe that's just me. That's not the important thing.

The important thing here is user friendliness. This situation is very likely to happen to users who are not package developers, and who have no idea where to even look for the relevant documentation about how to configure sone packaging system or another. After a reasonable time-out trying to download setuptools, please provide an error message specifying the exact problem: "cannot get an updated copy of setuptools from the Internet" and solution: "try using pip with --no-build-isolation".

[tfardet]

--extra-index-url is deterministic. The issue is it’s not what people think it is 🙂

I just ran it to test a new version of a package on test.pypi.
The first execution (incorrectly) installed the 2.7.0 package from pypi.
The second execution (correctly) installed the 2.7.1 package from test.pypi.

If it is deterministic, it sure doesn't look the part: I don't know what happens but it would be nice to have the latest version picked up consistently.
Until it does, @jaraco 's remark seems, if not technically correct, at least relevant in terms of not recommending a call that may not have the expected behavior.

[uranusjr]

Note that it takes time for test.pypi.org (and pypi.org, they’re the same) to propagate files over CDN, so a newly uploaded version may not be found by a package installer immediately. The example given above may just be an example for that.