bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes #31034

hroncok · 2022-01-31T14:23:58Z

Define posix_venv and nt_venv sysconfig installation schemes
to be used for bootstrapping new virtual environments.
Add venv sysconfig installation scheme to get the appropriate one of the above.
The schemes are identical to the pre-existing
posix_prefix and nt install schemes.
The venv module now uses the venv scheme to create new virtual environments
instead of hardcoding the paths depending only on the platform. Downstream
Python distributors customizing the posix_prefix or nt install
scheme in a way that is not compatible with the install scheme used in
virtual environments are encouraged not to customize the venv schemes.
When Python itself runs in a virtual environment,
sysconfig.get_default_scheme and
sysconfig.get_preferred_scheme with key="prefix" returns
venv.

https://bugs.python.org/issue45413

The scheme is used for bootstrapping new virtual environments. It is a copy of the "posix_prefix" or "nt" install scheme depending on the platform. The venv module now uses that scheme to create new virtual environments instead of hardcoding the paths depending only on the platform. Downstream Python distributors customizing the "posix_prefix" or "nt" install scheme in a way that is not compatible with the install scheme used in virtual environments are encouraged to supply a modified working "venv" scheme.

Lib/sysconfig.py

…hardcoded values

The tests have that one instead. This reverts commit 4686893.

FFY00

Thank you for looking into this! Though, I do have some concerns about if the approach taken was the correct one.

It is a copy of the "posix_prefix" or "nt" install scheme depending on the platform.

I think this is a bit messy, and not very nice for long term maintainability. There is also the existing issue of distributors patching the posix_prefix scheme, which would mess this up, and while that is not officially supported, it would be best to try to preemptively mitigate any issues there, as long as it is a reasonable option for CPython.

When I opened the issue, I envisioned adding a completely new scheme, and making the interpreter select those paths when we are in a virtual environment. We already have code to detect this (hint: pyvenv.cfg), so it should just be a matter of hard-coding the paths there.

This would also make the scheme be the same across operating systems, which is fairly neat IMO. prefix would be the virtual environment path, and everything builds from there.

That said, I would like both @jaraco and @zooba to take a look, if they are available. If not, I think it would be best to delay merging this for a bit, but I am not a core dev, so my opinion doesn't matter much there, if anyone is confident enough with it, go ahead and merge I guess 😅

Lib/venv/__init__.py

hroncok · 2022-02-01T20:51:37Z

We can certainly make get_prefered_scheme return "venv" in venvs. I don't know if it needs to be done as part of this change.

FFY00 · 2022-02-01T21:56:28Z

We can certainly make get_prefered_scheme return "venv" in venvs. I don't know if it needs to be done as part of this change.

Oh yeah, that is something I overlook in this preliminary review. I think we should definitely do that.

…cheme() returns "venv"

hroncok · 2022-02-01T23:06:23Z

We can certainly make get_prefered_scheme return "venv" in venvs. I don't know if it needs to be done as part of this change.

Oh yeah, that is something I overlook in this preliminary review. I think we should definitely do that.

OK, this is now done. But I am not sure if this is what I wanted :D

Lib/sysconfig.py

FFY00 · 2022-02-01T23:18:50Z

Thank you!

…venv is *venv*

encukou · 2022-03-16T14:41:48Z

Who else do should comment here before you can send the PR?

merwok · 2022-03-16T14:45:42Z

This needs a fix, so I'll merge the PR as it is.

This is confusing: why run to merge something before it’s complete?

encukou · 2022-03-16T14:52:52Z

This PR is complete now. There is discussion about how it can be made better, and that discussion is not really moving forward. Everyone is waiting for something while a fix is ready.
And the solution to this issue should get as much practical testing as it can before the API is locked in by 3.11.0 beta. At the pace it's been going, it won't get in at all.

merwok · 2022-03-16T15:47:28Z

Ah, I thought fix was referring to fixing an issue in this PR, but I see you mean fix for the ticket, i.e. the whole PR.
Didn’t understand that at first because I see this as a new feature!

encukou · 2022-03-16T16:14:57Z

Right, my wording was ambiguous.
It's a new feature for CPython, but it's an issue in software affected by the changes surrounding distutils deprecation/removal.

FFY00 · 2022-03-16T19:31:14Z

I think it is safe to say that the current state of the PR isn't clear to be the best approach. I don't understand the eagerness to merge this right now, I think that can do more harm than good.

The issue this PR solves already exists in the current released version, 3.10. We will not be able to fix it in that version, that ship has sailed. To work around this, we have already designed an approach to deal with this issue, which was adopted by the impacted parties (see pypa/virtualenv#2208).

This PR, which would land in 3.11, is incompatible with the adopted workaround, requiring more changes to the downstreams, and introducing more complexity in version compatibility code. If we merge it as-is and it gets released, we put more weight on the downstreams, and even more if it turns out that we need to change the approach later.

So, my proposal would be to drop get_venv_scheme and use a venv scheme directly instead in this PR, preserving compatibility with the downstream workaround.
We can then investigate adopting a new approach, and other needs that might arise. If we decide another approach is needed, we can then adopt it (honestly, I don't think it is), but please lets do not do that with an undereviewed one.

Most of my concerns with the current implementation, other than it being incompatible with the 3.10 workaround, are due to long term maintainability and distributors messing things up with patching, and can be solved just by changing the internal implementation, without breaking compatibility. We can handle them later.

FFY00 · 2022-03-16T19:37:42Z

It's a new feature for CPython, but it's an issue in software affected by the changes surrounding distutils deprecation/removal.

Not really. It is an issue in Python 3.10+ software, period. It was just introduced by changes to support the distutils deprecation, but this affect all Python 3.10 software that is on the business of manually creating virtual environments.

hroncok · 2022-03-16T19:46:37Z

So, the "venv" scheme would do what the function currently does, and the platform specific schemes would stay? I can do that as well.

FFY00 · 2022-03-16T19:51:05Z

Yes, exactly.

encukou · 2022-03-17T09:57:16Z

@zooba, what do you think about that?

hroncok · 2022-03-17T10:38:04Z

I've committed Drop get_venv_scheme() in favor of venv scheme. It can be reverted if @zooba does not like it.
There was some outdated documentation, so I've changed that in Apply my own suggestions from docs review.

I would squash those together, but I've been told not to push force.

Doc/library/sysconfig.rst

Doc/library/venv.rst

Doc/whatsnew/3.11.rst

zooba

Looks good to me. (Apologies for unresponsiveness - GitHub notifications are a bad way to reach me, unfortunately. bpo messages are far more reliable due to no noise.)

Doc/whatsnew/3.11.rst

Co-authored-by: Steve Dower <[email protected]>

FFY00

Thanks, this is significantly better IMO.

We can improve the implementation later if needed.

hroncok · 2022-03-18T10:01:54Z

encukou · 2022-03-18T10:05:02Z

Thanks for your work and patience, @hroncok and @FFY00!

Steve: OK, I'll keep it in mind. I was just drafting a direct email; hopefully that would work if notifications fail :)

zooba · 2022-03-18T12:17:23Z

@encukou I suspect direct emails are going to become important for me once all notifications move to GitHub. I can only see things getting even worse than they are now 😞

As a result of python/cpython#31034, venv now calls sysconfig.get_paths. This messes up the mock call count. The change to using Signature.bind isn't strictly necessary, but fixes the initial symptom (since venv calls sysconfig.get_paths without kwargs). This also means we can use sysconfig.get_paths however we want.

* Fix mock in test_executable_missing_post_creation for Python 3.11 As a result of python/cpython#31034, venv now calls sysconfig.get_paths. This messes up the mock call count. The change to using Signature.bind isn't strictly necessary, but fixes the initial symptom (since venv calls sysconfig.get_paths without kwargs). This also means we can use sysconfig.get_paths however we want. * add comment, unskip on pypy3 * mock venv.EnvBuilder.create * remove unused imports Authored-by: hauntsaninja <>

hroncok requested a review from vsajip as a code owner January 31, 2022 14:23

the-knights-who-say-ni added the CLA signed label Jan 31, 2022

bedevere-bot added the awaiting review label Jan 31, 2022

hroncok marked this pull request as draft January 31, 2022 14:24

hroncok added 2 commits January 31, 2022 15:54

Use the same conditional as previously in venv

4686893

Use os.sep

6e98457

hroncok mentioned this pull request Jan 31, 2022

RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing pypa/build#215

Closed

hroncok commented Feb 1, 2022

View reviewed changes

Lib/sysconfig.py Outdated Show resolved Hide resolved

A bit better wording of the comment

602c2ad

hroncok force-pushed the venv_scheme branch from eb440bf to 602c2ad Compare February 1, 2022 18:33

hroncok added 2 commits February 1, 2022 20:04

Add a test that the venv scheme does not regress from the previously …

44f588a

…hardcoded values

Revert "Use the same conditional as previously in venv"

8af07ee

The tests have that one instead. This reverts commit 4686893.

hroncok marked this pull request as ready for review February 1, 2022 19:32

FFY00 reviewed Feb 1, 2022

View reviewed changes

Lib/venv/__init__.py Outdated Show resolved Hide resolved

hroncok added 2 commits February 1, 2022 23:54

venv: Resolve paths with env_dir as all bases

c734658

Optional: When we are in virtual environment, sysconfig.get_default_s…

0a8556d

…cheme() returns "venv"

FFY00 reviewed Feb 1, 2022

View reviewed changes

Lib/sysconfig.py Outdated Show resolved Hide resolved

hroncok added 3 commits February 2, 2022 01:05

Also return venv from get_preferred_scheme()

cb2ed5b

Rename binname to binpath in test for consistency with other names

af47953

Add documentation and what's new

4cc4015

hroncok force-pushed the venv_scheme branch from be612eb to 4cc4015 Compare February 2, 2022 10:46

hroncok added 4 commits February 2, 2022 13:05

Also mention this change in What's new for venv

5d04c22

Sysconfig docs: The venv scheme is also for running virtual environments

5f35cd8

Typo

e7444a2

Assert the preferred(prefix) and default sysconfig install scheme in …

30fdee3

…venv is *venv*

Drop get_venv_scheme() in favor of *venv* scheme

283db7a

hroncok commented Mar 17, 2022

View reviewed changes

Apply my own suggestions from docs review

594c4d9

zooba reviewed Mar 17, 2022

View reviewed changes

Doc/whatsnew/3.11.rst Outdated Show resolved Hide resolved

Typo

9890918

Co-authored-by: Steve Dower <[email protected]>

hroncok changed the title ~~bpo-45413: Define "posix_venv" and "nt_venv" sysconfig installation schemes~~ bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes Mar 17, 2022

FFY00 approved these changes Mar 17, 2022

View reviewed changes

encukou merged commit 48d9262 into python:main Mar 18, 2022

bedevere-bot removed the awaiting merge label Mar 18, 2022

hroncok deleted the venv_scheme branch March 18, 2022 10:01

hauntsaninja mentioned this pull request Apr 16, 2022

Fix test_executable_missing_post_creation for Python 3.11, PyPy pypa/build#459

Merged

yvt mentioned this pull request Sep 19, 2022

Failure in Windows and MacOS CI servo/rust-mozjs#559

Open

FFY00 mentioned this pull request Dec 7, 2024

venv.EnvBuilder.ensure_directories returns incorrect inc_path value #127724

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes #31034

bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes #31034

hroncok commented Jan 31, 2022 •

edited

Loading

FFY00 left a comment •

edited

Loading

hroncok commented Feb 1, 2022

FFY00 commented Feb 1, 2022

hroncok commented Feb 1, 2022

FFY00 commented Feb 1, 2022

encukou commented Mar 16, 2022

merwok commented Mar 16, 2022

encukou commented Mar 16, 2022 •

edited

Loading

merwok commented Mar 16, 2022

encukou commented Mar 16, 2022

FFY00 commented Mar 16, 2022 •

edited

Loading

FFY00 commented Mar 16, 2022 •

edited

Loading

hroncok commented Mar 16, 2022

FFY00 commented Mar 16, 2022

encukou commented Mar 17, 2022

hroncok commented Mar 17, 2022 •

edited

Loading

zooba left a comment

FFY00 left a comment

hroncok commented Mar 18, 2022

encukou commented Mar 18, 2022

zooba commented Mar 18, 2022

bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes #31034

bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig installation schemes #31034

Conversation

hroncok commented Jan 31, 2022 • edited Loading

FFY00 left a comment • edited Loading

Choose a reason for hiding this comment

hroncok commented Feb 1, 2022

FFY00 commented Feb 1, 2022

hroncok commented Feb 1, 2022

FFY00 commented Feb 1, 2022

encukou commented Mar 16, 2022

merwok commented Mar 16, 2022

encukou commented Mar 16, 2022 • edited Loading

merwok commented Mar 16, 2022

encukou commented Mar 16, 2022

FFY00 commented Mar 16, 2022 • edited Loading

FFY00 commented Mar 16, 2022 • edited Loading

hroncok commented Mar 16, 2022

FFY00 commented Mar 16, 2022

encukou commented Mar 17, 2022

hroncok commented Mar 17, 2022 • edited Loading

zooba left a comment

Choose a reason for hiding this comment

FFY00 left a comment

Choose a reason for hiding this comment

hroncok commented Mar 18, 2022

encukou commented Mar 18, 2022

zooba commented Mar 18, 2022

hroncok commented Jan 31, 2022 •

edited

Loading

FFY00 left a comment •

edited

Loading

encukou commented Mar 16, 2022 •

edited

Loading

FFY00 commented Mar 16, 2022 •

edited

Loading

FFY00 commented Mar 16, 2022 •

edited

Loading

hroncok commented Mar 17, 2022 •

edited

Loading