Unexpected --user behavior since version 23.1

[flowmeadow]

Description

Hey everyone,
I'm using pip typically in virtual environments (managed by PyCharm). Since version 23.1 I get an error message when installing packages.

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

A little research told me, that a --user flag within a venv does not really make sense. However, it seems that some kind of default --user is set since 23.1.

A workaround I found is to add the --no-user flag.

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy --no-user
Collecting numpy
   Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2

For completeness, downgrading to 23.0.1 with the --no-user flag gives the me the behavior I'm used to.

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>python.exe -m pip install --upgrade pip==23.0.1 --no-user
Collecting pip==23.0.1
  Using cached pip-23.0.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.1
    Uninstalling pip-23.1:
      Successfully uninstalled pip-23.1
Successfully installed pip-23.0.1

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
Collecting numpy
  Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2

[notice] A new release of pip is available: 23.0.1 -> 23.1
[notice] To update, run: python.exe -m pip install --upgrade pip

Not sure if this is a bug or a feature. Maybe I'm also missing something very obvious. But at least it is documented now for anyone experiencing the same issue :)

Expected behavior

No response

pip version

23.1

Python version

3.9

OS

Windows 10

How to Reproduce

create a venv with pip 23.1
try to install a package with pip install <any-package-name>

Output

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Pip 23.1 does not have a default to --user. However it's possible you are picking up a user config of some sort. Please can you provide the output of pip config debug in both the failing and the successful cases?

[flowmeadow]

Sure, thanks in advance.

Pip 23.0.1 (successful case):

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip config debug
env_var:
env:
global:
  C:\ProgramData\pip\pip.ini, exists: False
site:
  C:\Users\florian.wiese\PycharmProjects\GUIManager\venv\pip.ini, exists: False
user:
  C:\Users\florian.wiese\pip\pip.ini, exists: False
  C:\Users\florian.wiese\AppData\Roaming\pip\pip.ini, exists: False

Pip 23.1 (Failing case):

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip config debug
env_var:
base:
  C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\pip.ini, exists: True
    install.user: yes
    uninstall.user: yes
    freeze.user: yes
    list.user: yes
env:
global:
  C:\ProgramData\pip\pip.ini, exists: False
site:
  C:\Users\florian.wiese\PycharmProjects\GUIManager\venv\pip.ini, exists: False
user:
  C:\Users\florian.wiese\pip\pip.ini, exists: False
  C:\Users\florian.wiese\AppData\Roaming\pip\pip.ini, exists: False

[pfmoore]

So that's your issue. In the failing case, you have a base config that sets user to true. This looks like the Windows Store installation of Python. I'm not sure why they set a pip config (I think it's a workaround for a permission issue with Windows Store apps) but IMO setting user globally like this is a bad idea. But it's what we have to deal with, unfortunately.

This was caused by

In the case of virtual environments, configuration files are now also included from the base installation. (#9752)

That issue was extensively discussed, but it appears that the case of the Windows Store distribution wasn't known (or wasn't considered). IMO this is a sufficiently serious problem that it requires a bugfix. I'm not sure it's possible to simply revert the change (reverting a merge commit is a bit beyond my git skills!) so we'll need someone to work on a fix ASAP. @pelson as the author of the original change, are you able to look at this? Alternatively, if anyone can tell me the right git incantation to revert #11487, I can do that and we can revisit the feature for a future pip version.

Thanks for reporting this @flowmeadow!

[pfmoore]

As a quick proof of concept, I've created #11983 to patch out the BASE config. This might form the basis of a minimal fix if we can't find a better solution.

[pfmoore]

I also opened python/cpython#103646 to discuss with CPython whether there's a better way than a pre-installed pip.ini to achieve what they are trying to do with the Store installation.

[flowmeadow]

Thanks a lot for the analysis. I can confirm that its the Windows Store installation. I remember having a very bad feeling using it. Well, at least this was the first issue that arose in almost two years.

[pfmoore]

Generally, I think the Store install should be fine. There's some quirks like this, but they mostly seem to work. And having people use the distribution and report issues like you have is the only way we'll know what needs improving, so your help is appreciated.

[abhinavsingh]

jFYI, this is also impacting proxy.py workflows e.g. https://github.com/abhinavsingh/proxy.py/actions/runs/4717368971/jobs/8450593742

[pfmoore]

@abhinavsingh that's nothing to do with this issue. That's the deprecation of pkg_resources, which is #11975 (and which is now fixed in main, and awaiting a bugfix release which will come once we've established what fixes we need to include).

[pfmoore]

By the way @flowmeadow, as a temporary workaround, you should be able to set the user option to false in a config file that's higher priority than the "base" one (or just set $PIP_USER=0 in your environment).

[notatallshaw]

So that's your issue. In the failing case, you have a base config that sets user to true. This looks like the Windows Store installation of Python. I'm not sure why they set a pip config (I think it's a workaround for a permission issue with Windows Store apps) but IMO setting user globally like this is a bad idea

My understanding is that this is the model of Windows Store installs, the base install is in a global space that any specific user may not have permissions to modify, therefore the application must store user specific data in a user space.

But unlike Linux distros which manage the whole Python pipeline for the system install such as installing Python packages like Pandas using their own distribution method outside Pip there is no such infrastructure on the Windows Store and users are expected to use Pip install.

Maybe a Windows specific exception will need to be made here as it's model of users is not a 1-1 mapping to Unix-like model of users and root anyway.

I'm going to ping @zooba just in case he wants to comment here (and maybe correct any misunderstandings).

I do wishfully also wonder if the default behavior of installing Python on Windows could in the future switch from the the Windows Store to winget and behave more like Linux distros? But I'm throwing that idea out with no particular understanding of the problems, so it may be a faulty idea.

[zooba]

switch from the the Windows Store to winget and behave more like Linux distros?

Show me a Linux distro that downloads our installer and runs it with unreliable command line options and I'll consider it (spoiler: no I won't 😉 winget is not a good way to manage Python installs, I'm afraid).

just in case he wants to comment here (and maybe correct any misunderstandings)

I've put plenty of explanation on the CPython bug already. My fundamental disagreement is with configurations located in sys.base_prefix being used when sys.base_prefix != sys.prefix, as it makes it impossible to use a sys.prefix configuration in something that may be used as a base environment.

Perhaps sys.base_prefix configuration files should use a different filename, or different section names or something. Or some extra setting to say "please consider the settings in this file for venvs created from this environment".

I can imagine useful scenarios for files in both sys.base_prefix and also sys.prefix when ==sys.base_prefix, but they all assume the distributor knows what you should use by default (e.g. platform name, index-url, etc.). If pip wants to say that distributors don't know the right install locations, that's fine, we distributors will just patch source files instead of adding overridable config settings.

[pfmoore]

My fundamental disagreement is with configurations located in sys.base_prefix being used when sys.base_prefix != sys.prefix, as it makes it impossible to use a sys.prefix configuration in something that may be used as a base environment.

To be clear, there's no such thing as a "sys.prefix configuration" as you describe it. If you look in the documentation, the location {sys.prefix}/pip.ini isn't mentioned as a source of config data, although in the code, it's used as the location of the "site" data when you're not in a virtual environment. I think that's probably just a problem with the docs, though, and what you mean here is the "site" location.

I do agree that it's wrong to have the "base" config come from the same place as the "site" config of the system Python, precisely because as you say it means that the system site config "leaks" into a virtual environment. I think that's a flaw in the design of #9752 which we should fix¹.

As a general point, I personally don't consider pip's configuration files as intended for distributors - they are for users and site admins to manage. Distributors shouldn't be using them, and very definitely shouldn't be relying on what's in them. If distributors need configuration options, they should discuss that with the pip maintainers and we can see how best to address such a need. But that's just my personal view - I'd be interested if the other @pypa/pip-committers agree with me on that.

If pip wants to say that distributors don't know the right install locations, that's fine, we distributors will just patch source files instead of adding overridable config settings.

I think that's an interesting use case that we should discuss further. It may be a reasonable use for certain types of config. One key difference if a distributor patches source files is that the support burden is squarely on the distributor in that case. One problem with diagnosing this issue was that it took a while to realise that the offending user config wasn't actually set by the user, and the user didn't know about it.

Footnotes

1. As a minor point, I'm not keen on having a different basename for the file - knowing that pip's config file is always called pip.ini is a useful consistency for people trying to work out what's going on. ↩

[omarirfa]

Same issue here, for now I just reverted back to pip 21.3 and everything worked again. Although, in my case both pip 23.0.1 and pip 23.1 are failing with the same error

[omarirfa]

By the way @flowmeadow, as a temporary workaround, you should be able to set the user option to false in a config file that's higher priority than the "base" one (or just set $PIP_USER=0 in your environment).

For solution 1 how do u find a file that is higher priority than the base file? I was wondering if its one of these files I will have to change?

If it is one of the user or global files, I have tried adding this to the pip.ini, but it did not work:

[global]
user=false

For solution 2, I tried using $PIP_USER=0 while in the venv it gives an error saying it is not a recognized internal or external command. Is there a specific way to set or run this?

[pradyunsg]

@pfmoore How do you feel about reverting and reopening #9752, and we can come back around to re-adding that in a follow up change for 23.2?

[pfmoore]

@pradyunsg As I said above I'd be happy to do that but I don't know how (in git) to revert a merge commit. If you can confirm how I do that I'd be far happier with that approach.

On the longer term resolution of #9752, I'd point out that this problem has reinforced my dislike of having the "base" environment leak into virtual environments like this, so I'd actually oppose #9752 in its stated form. I'd like a much better justification of why the global setting isn't sufficient for the original use case, and assuming the change can be justified, I'd want the "base" config to be very explicitly not the same as the "site" config of the base environment. So I think there's more than just "re-adding" the feature involved here, more like "re-evaluating" and "re-designing" it in a way that accounts for the way the Windows Store distribution uses the existing config mechanism.

[pradyunsg]

Whoop, I missed that while skimming the issue! Sorry about that!

git revert -m 1 <hash>

(at PyCon US, so my replies might be slow!)

[pfmoore]

Ah! I tried that, but was confused that there was no reversion of the news entry. But I guess that makes sense, as it was merged into the changelog when I did the release. Thanks for confirming.

#11987

[layday]

@pfmoore FYI, you reopened #9752, but it's still locked for comments - not sure if that was intentional.

[pfmoore]

Thanks for the heads up - I'll fix that!