Don't pip install without user knowledge and allow to opt out

[FFY00]

It seems like pep517.build and pep517.check will run pip install on their own without ever telling anything to the user, and there's no way to disable this.

Linux distributions often need a tool that simply builds a wheel, one that does not go run things like pip install on our backs. A lot of the reasons have been outlined in pypa/setuptools#2080. The problem is not just it is using pip, it's that it is fetching sources from the internet and installing packages without I ever tell it to do it. When I make a distribution package I want to be the one fetching and verifying all sources, and I want to be in control of what is installed to the filesystem.

All I need is a simple build system without many dependencies (because we will need to bootstrap those, which is a lot of work). It should not do any package management, just build what I ask it to. It will fail if there are missing build dependencies.

And I believe this should be the default behavior, I think python -m pep517.* should be a build system first (actually, I think it should only be a build system but I don't think you agree). If a user wants to install a project from source, pip works, but please let's not add pip as a dependency at this level. If you really want pip to be run I would suggest adding a command-line option that would enable that behavior.

Right now python -m pep517.build will error out because pip is not installed:

/usr/bin/python: No module named pip
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.8/site-packages/pep517/build.py", line 124, in <module>
    main(parser.parse_args())
  File "/usr/lib/python3.8/site-packages/pep517/build.py", line 120, in main
    build(args.source_dir, dist, args.out_dir)
  File "/usr/lib/python3.8/site-packages/pep517/build.py", line 87, in build
    env.pip_install(system['requires'])
  File "/usr/lib/python3.8/site-packages/pep517/envbuild.py", line 100, in pip_install
    check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--ignore-installed', '--prefix', '/tmp/pep517-build-env-v568jxh0', 'flit_core >=2,<3']' returned non-zero exit status 1.

There is no reason for pip to be there if I have every build dependency already.

Just a note on the shape python packaging is taking. I believe everything here should be modular, having a build system and installer, and then a package manager. I think pip should offload the work to these tools and just deal it the package management (fetching sources, managing dependencies, etc.).

[eli-schwartz]

This is an absolute blocker for any future distribution acceptance of pep517 in the linux packaging environment (speaking only for archlinux here, but I'd imagine other distros will feel the same). If pep517.build requires pip, we won't use pep517.build...

[eli-schwartz]

Hmm, looks like in theory we could write our own tool which wraps around Pep517HookCaller just in order to skip this, but then that begs the question "why pep517 at all" which seems kind of meh.

[FFY00]

Yes, we could, it would be relatively simple. The thing here is that I think it should be provided by the upstream. I don't see any reason why distributions should need to maintain their own python packaging stack.

[pfmoore]

Why is pip wheel --no-build-isolation --no-deps <source dir> not suitable, in that case? If you don't want the tool to manage the build environment (i.e., to pip install build requirements) then I'm not clear why pip wheel --no-build-isolation --no-deps doesn't work for you.

[FFY00]

See the issue I linked. TLDR: pip has too many dependencies, we would need to bootstrap every single one of them.

I still fail to understand why is pip install needed in pep517.build, you could do the same with pip.

[eli-schwartz]

Pip itself is not acceptable for reasons elaborated upon in the linked setuptools issue.

[pfmoore]

pep517.build is setting up an isolated build environment, so it sets up dependencies (using pip, because it's a very simple example tool, and doesn't pretend to offer a full-featured range of ways to manage the build environment, so it goes with the standard Python tool which is pip). If you don't want or need that, then just use the hooks module and manage the build environment yourself, as suggested above. pep517 is meant to be a low-level library that you build tools on top of, so that's the expected use case here.

[takluyver]

Maybe it was a mistake to make the high-level pep517.build and pep517.check interfaces at all. The focus of pep517 is an implementation of what the PEP 517 spec describes: running hooks in a subprocess, handling fallbacks when an optional hook isn't implemented, and reporting certain likely problems as clear exceptions.

pep517.build implements essentially the simplest possible frontend tool, to exercise and illustrate how the core functionality can be used. Probably every serious frontend would write its own equivalent of that layer rather than using it from pep517.

[takluyver]

(I'll try to get round to documenting this expectation more clearly)

[takluyver]

I think it should be provided by the upstream

Specifically, what, though? I assume distros want to handle dependencies entirely through their own package systems - we're obviously not going to abstract over apt/yum/conda/etc., and I doubt you'd use it if we did. Once you've got all the build dependencies installed, you can pretty much just call Pep517HookCaller.build_wheel(), and that's the end point of PEP 517. There's a separate low-level installer tool under discussion here.

[FFY00]

Specifically, what, though?

A simple interface to just build wheels.

I assume distros want to handle dependencies entirely through their own package systems - we're obviously not going to abstract over apt/yum/conda/etc., and I doubt you'd use it if we did.

I don't want it to touch package management at all. If a needed dependency is missing it just errors out. It's a build system...

Once you've got all the build dependencies installed, you can pretty much just call Pep517HookCaller.build_wheel(), and that's the end point of PEP 517.

Yes, I just want this -- parse pyproject.toml and call Pep517HookCaller.build_wheel() with the correct options. Why do I need to install another tool to do just this?

But I guess if you don't want it here, I can maintain it separately.

There's a separate low-level installer tool under discussion here.

I am aware.

[pfmoore]

Yes, I just want this -- parse pyproject.toml and call Pep517HookCaller.build_wheel() with the correct options. Why do I need to install another tool to do just this?

I guess the answer to that is "because that functionality doesn't exist in this library (yet, and maybe never will)".

@takluyver It does feel like there's some restructuring we should do here, as the current situation is clearly setting people's expectations incorrectly 🙁 I've raised #82 to discuss this.

[FFY00]

I think the easiest way to fix this is to not ship any commands, make them just examples.

[eli-schwartz]

I had pondered trying to submit a patch to make pip_install() only invoke pip if the build requires were not available, but then I hit the snag of I cannot figure out a way to check a requirement specifier to see if it exists, other than apparently to use a pkg_resources WorkingSet (code soup). importlib.metadata can take a name and produce a found distribution and its version, but how do I compare this...

[takluyver]

parse pyproject.toml and call Pep517HookCaller.build_wheel() with the correct options

This is pretty trivial in Python code, so I guess you mean you want a little wrapper to do it from a shell script? I wouldn't be against putting something at that level into pep517.

[pradyunsg]

With pypa/build on it's way, I think the underlying issue here will be resolved. And, #82 probably captures the rest of the relevant parts of this issue.

Closing!