Add "python -m setuptools" invocation method

[eli-schwartz]

Summary of changes

This fully replaces the use of "python setup.py " with the equivalent "python -m setuptools ". In the process, we guarantee that a setup.py which imported only distutils will still get the enhanced setuptools treatment previously provided only by pip install.

This opens the way for future improvements to setuptools, such as validating setup_requires (if listed in setup.cfg) before invoking the setup() function and running code.

It is also the only way to invoke setuptools to build and install packages which fully moved to setup.cfg and no longer provide a stub setup.py with the content:

from setuptool import setup
setup()

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d. See documentation for details

[jaraco]

I'm disinclined to accept this change, mainly because it deviates from the recommended practice (using pep 517 to build) and because it adds complexity to the codebase. setup_requires needs to be deprecated. If you want a command to build setuptools-based projects, consider using pep517 with python -m pep517.build .. Give that a try and let me know if it doesn't fully address your needs.

[eli-schwartz]

My needs are installing such projects, so a tool that only provides a "build" command cannot work.

And I was hoping to reduce the confusion in the user experience, not add to it. I'm involved in distro packaging, so we need to be able to build from source, bootstrap setuptools with its dependencies, and build 1000+ more packages in a somewhat coherent manner. Currently we have a simple guideline: python setup.py build && python setup.py install --root="${packaging_root}". It works for distutils projects (such as the packages setuptools itself depends on), it works for setuptools projects, it works to bootstrap setuptools after running python bootstrap.py, and it works for everything else, because non-setuptools projects invariably turn out to be sub-quality boutique formats for generating setup.py because they have no user story (looking at poetry, which can either install to some venv in $HOME/.cache after 15 minutes "calculating..." or generate a setup.py and stick it in a sdist).

I'd like to replace that with "python setup.py ... for distutils projects, python -m setuptools ... for setuptools projects, please consider moving non-bootstrap distutils projects to use setuptools anyway as no one should really be using that". It's also convenient that they would both provide the same command-line options.

What I don't want is for packagers to have to think about every single package, decide whether it is a pep517 compliant package in which case to use one uniquely distinct program to build it (pep517.build) and another uniquely distinct program to install it (I am still in the dark as to what this is supposed to be), or a non-pep517 compliant program in which case "carry on with what's worked perfectly well in the past".

I definitely don't want packagers to have to do a mad scramble when setup.py install which works everywhere, suddenly doesn't work for 2 out of 1000 packages, have no idea what to do, and then, after a lot of confusion and inability to package the project, settle on:

python -c "from setuptools import setup; setup();" install --root="${packaging_root}"

while simultaneously complaining and holding this up as a prime example of the nonstop, headlong rush towards confusion and "some project removed setup.py and replaced it with pip, @eli-schwartz please help me what do I do".

As a matter of curiosity, how is pep517.build intended to improve packages which currently have sub-par metadata due to using straight from distutils.core import setup; setup(...)? I would sort of like to get the benefits of this too, but currently I'm not aware of any convenient method to do so other than once again "replace everything with pip". It's not like these projects will have a pyproject.toml...

(Which leads me to the question: do you actually really intend to completely remove setup.py install functionality? How do you intend to handle the many projects still on PyPI which have setup.py using the historic, de facto standard of setuptools, which will never be updated and hence also won't be updated to have a pep517 pyproject.toml? Will you hold a mass deletion party one day where you unceremoniously give them all the boot?)

[eli-schwartz]

I personally don't care even the slightest bit about setup_requires, since we require installing all install_requires via side channels and usually our only interest in setup_requires is patching them out because we really don't care about upstream's desire to misuse setup_requires for installing https://pypi.org/project/invoke/ on the release manager's machine. I did think it would be a neat bonus for you, though. Apologies if I misunderstood previous issues.

[pganssle]

My needs are installing such projects, so a tool that only provides a "build" command cannot work.

You should use pip install for this. If you need to handle build-time dependencies, you should use a PEP 517-compatible installation tool like pip.

[eli-schwartz]

We're definitely not using pip to install pip or its dependencies, that would be an irreconcilable bootstrap violation. And I just got finished explaining that my interest is distro packaging and we don't need anyone's help to handle dependencies at all, so I'm confused why it's being brought up again.

In fact, it's utterly forbidden in the strongest possible terms to handle build dependencies like that. It's required to convert build dependencies into an externally managed distribution package manager build graph, and build each member in isolation using non-python tools to manage dependency resolution, chroot containers, and build steps (the build steps will, at last, use a python tool).

[pganssle]

@eli-schwartz First of all, I think I should say thank you for putting in the effort to do this, and for explaining your use case. I do think that we're very unlikely to accept this change.

I myself suggested something similar in issue #1515, or at least opened the discussion for a path forward that does not involve invoking setup.py. I have since come to believe that while something like a generic build front-end is useful and necessary, setuptools is not that front-end.

Regarding this:

We're definitely not using pip to install pip or its dependencies, that would be an irreconcilable bootstrap violation. And I just got finished explaining that my interest is distro packaging and we don't need anyone's help to handle dependencies at all, so I'm confused why it's being brought up again.

No one says you have to install pip with pip. Though you don't really need to install pip, because it comes bundled with Python in Python 3, and python -m setuptools won't help you in Python 2.7 because we've dropped support for Python 2.

With regards to the change in general, the reason we think this is going in the wrong direction is that for a long time there was pretty strong "vertical integration" in build tools - your build tool did the installation for you. With PEP 517, we've moved to a more modular system with standard and well-defined roles for build backends and build front-ends, which makes things much easier for tools like tox, pip and distro packagers, because in a PEP 517/518 world, they all just have to code against the PEP 517 standard to build wheels, and then they can take wheels (a standardized format) and install them however they desire.

Previously, setuptools provided both a library for building projects and an extensible command line interface exposed in setup.py. We've decided that maintaining the CLI is no longer feasible with the resources and legacy baggage that we have, and we're getting out of that business. It's why we've actively removed setup.py upload and why we've deprecated setup.py test, and it's also why we will actively remove setup.py install and, eventually, all other commands.

Adding a way to invoke setuptools directly via the command line is, in my opinion, a step in the wrong direction there, because it encourages behavior that we're actively trying to discourage. Sorry if this is disappointing to you.

[eli-schwartz]

Though you don't really need to install pip, because it comes bundled with Python in Python 3

Hopefully it won't surprise you to learn that Linux distributions build pip from source rather than using vendor blobs in python? One advantage of this is that we push users to always have the the latest version of pip... no one on Arch Linux has ever had to install python+pip, then run python -m pip install -U pip in order to get support for the latest standards which aren't available in the original Python release.

With PEP 517, we've moved to a more modular system with standard and well-defined roles for build backends and build front-ends, which makes things much easier for tools like tox, pip and distro packagers, because in a PEP 517/518 world, they all just have to code against the PEP 517 standard to build wheels, and then they can take wheels (a standardized format) and install them however they desire.

That's all fine and well if you have a replacement to point people to. I'm living in a world where I don't have a replacement for setup.py install, and you're telling me "just use pip for everything". I don't want a non-modular tool with extremely sharp edges such as downloading and installing dependencies I haven't authorized, attempting to remove older versions from /usr/lib/python3.8/site-packages when I've specified a custom --root, respecting the --user flag in $HOME/.config/pip/pip.conf which is intended for controlling how the user's interactive pip experience operates, not how my isolated package building process creates a distro package, etc. etc. pip is always something to fight against and wrangle into submission by hammering it with options.

pip also builds in mysterious randomly generated directories in /tmp/ and embeds non-reproducible binary output based on these directory paths.

pip is also 24 different packages which need to be bootstrapped (because we devendor it, and this has so far worked pretty darn well), and in order to build manual pages we also need sphinx (26 packages partially overlapping) as a build-only dependency, so there's another cascade of packages for a grand total of 50.

50 packages before we can use the humongous footprint of pip to emulate 'setup.py install'. As you said, we're moving to a more modular system.

...

tl;dr I would greatly appreciate it if there were some option other than pip for installing pep517 wheels, and I believe that until that happens, it is reasonable to consider setuptools install as a transition tool. Since we're not ready to get rid of setup.py install anyway, it seems like a practical win, rather than an ideological loss, to consolidate the scene a bit by at least encouraging everyone using setuptools's install to at least do so with one codepath. You could also get people used to the idea of running a program ("the setuptools program") instead of executing a setup.py file. This is sort of kind of the case with pip, if pip was something everyone was happy to use. Eventually, tell them to use a different program, because "we discontinued setuptools, the new one is lots better".

This PR was intended to solve that by moving some code buried in the guts of setuptools.build_meta:_BuildMetaBackend as a class method, and making it usable to consumers. Though I'm not entirely sure what it's trying to do there, I do think it would be nice to make it rely on a single run_setup function.

It could also feasibly obsolete this: https://github.com/pypa/pip/blob/9cbe8fbdd0a1bd1bd4e483c9c0a556e9910ef8bb/src/pip/_internal/utils/setuptools_build.py#L14-L47

[pganssle]

Hopefully it won't surprise you to learn that Linux distributions build pip from source rather than using vendor blobs in python?

The "vendor blob" is a zip file containing the source code of pip, it's pretty easily validated. I think you'll find that I personally find the "we must build everything from our definition of source" to be quite cargo-culty, so this will get very little traction from me.

tl;dr I would greatly appreciate it if there were some option other than pip for installing pep517 wheels, and I believe that until that happens, it is reasonable to consider setuptools install as a transition tool.

You're welcome to consider it that, but it's an unsupported path and the maintainers of setuptools do not consider it to be so. If you're unwilling to build something around pip (even if you have to do something special to "build pip from source", and then use pip to build everything else), then I suppose the best solution for you is to write your own wheel installer? I don't think it's terribly difficult. Other wheel installers might even exist.

In any case, our policy here for years has been that if someone opens a bug that can be solved by switching from setup.py install to pip install, we generally do not fix that bug. If we add a python -m setuptools tool here, it will be very confusing messaging, because we'll say, "Oh you aren't supposed to use python -m setuptools, we just added it because some distributors use unsupported pathways."

It's also trivial to write your own wrapper that just does this and maintain it outside of setuptools, so if you insist on using the unsupported install mechanism (that we plan to actively remove as soon as possible), then I recommend writing a command line utility of your own that invokes setup.py for you, I guess?

[pganssle]

This PR was intended to solve that by moving some code buried in the guts of setuptools.build_meta:_BuildMetaBackend as a class method, and making it usable to consumers. Though I'm not entirely sure what it's trying to do there, I do think it would be nice to make it rely on a single run_setup function.

Oh, I should note that setuptools.build_meta is intended to exist only as a PEP 517 backend. Anything not exposed is done deliberately so. Invoking a PEP 517 build is actually more complicated than just calling the methods, there are requirements about the build environment, etc: https://www.python.org/dev/peps/pep-0517/

If you really don't want to use pip, it shouldn't be too hard to write a pep517 frontend that behaves the way you want.

[eli-schwartz]

The "vendor blob" is a zip file containing the source code of pip, it's pretty easily validated. I think you'll find that I personally find the "we must build everything from our definition of source" to be quite cargo-culty, so this will get very little traction from me.

I'm talking about "vendor" blobs, and you're attempting to argue the counterpoint that it isn't a "binary" blob because it can be validated. Nice attempt at dodging, I guess?

The disconnect here is sufficient that I'm not sure why I expected to have a rational conversation. We're not even talking about the same topic, it seems.

Nevertheless, I will try. So here you go: https://pip.pypa.io/en/stable/development/vendoring-policy/ This policy violates our policy, and therefore we rm -rf pip/_vendor/ (except for __init__.py) and use non-bundled, system modules, which have various benefits easily recognizable to anyone who has ever done anything in the linux distributions field, including, for example, the fact that it doesn't trample all over the system SSL policy.

The existence of the pip/_vendor/ directory is the commonly accepted meaning of "vendored" code, and until today, I thought it was the only meaning.

As a matter of curiosity, can you point me to some sort of reference for your alternative definition?

[pganssle]

I'm talking about "vendor" blobs, and you're attempting to argue the counterpoint that it isn't a "binary" blob because it can be validated. Nice attempt at dodging, I guess?

I would ask that you assume good faith here, as it's hard to remain constructive in a hostile and adversarial discussion.

One can see two problems with "vendored blobs", one is that it's vendored (and thus changes cannot be centralized) and the other is that it's a blob (and thus you cannot build it from source). I mistakenly thought that you were concerned with the "blobs" part, which is a common issue with people who do not want to use wheels and prefer to "build from source", even pure Python wheels (which are just zip files containing the source code).

If you're more concerned about the "vendored" part, then yes, wheel vs. sdist is not really a concern.

In any case, I'm going to go ahead and close this, because unfortunately we simply do not have the resources to maintain a setuptools-specific installer, particularly when the one we have is horribly broken in many circumstances.

This issue seems to be popular lately, and the discussion of what, if anything, will be done to allow downstream distributors to avoid using pip can be discussed in #2088.

[eli-schwartz]

I would ask that you assume good faith here, as it's hard to remain constructive in a hostile and adversarial discussion.

I apologize for losing my temper.

people who do not want to use wheels and prefer to "build from source", even pure Python wheels (which are just zip files containing the source code).

This much, I don't have strong feelings about. I think using wheels can be workable, at least depending on whether any patches need to be applied (backports, system integration fixes, etc.). Another issue is that we typically run unittests to ensure the package works in our environment, and while we can often (but not always) get tests from an sdist, wheels... not so much. So it might only be practical for 1) pure-python wheels, 2) whose developers don't have unittests anyway.

As pep517 is fairly easy to bootstrap (setuptools+friends, and toml), we can probably use that to generate wheels from sources in such cases anyway, the problem then gets back to "how to install them", which is where I'm currently baffled...

This issue seems to be popular lately, and the discussion of what, if anything, will be done to allow downstream distributors to avoid using pip can be discussed in #2088.

Hmm, well, I will hope for some movement there, I suppose...