Virtualenv does not work correctly on python 2.7 from python.org and new macOS 12 Monterey #2284
Comments
Feel free to put in a PR to fix it. 👍 |
|
@gaborbernat, can you help me to find ideas how to find the reason, why this happening? |
|
Sorry, I do not have time for that at this point in time. |
|
@gaborbernat, ok, thank you. |
|
Hello, we are having the same issue. It only happens with python2.7 installed from python.org. Seems like virtualenv is not creating its own copy of python and pip instead using the global one installed under /Library/Frameworks/Python.framework/Versions/2.7/. |
|
Step one would be to run the test suite against that python and see what breaks 🤔 and then hunt the problems down one by one. |
|
I tried however there are too many errors and going so deep, I think I'm not making any progress. Here is the log for who picks this up. |
|
For what it is worth it might look like a lot but there's a high degree of duplication in between there. So you might just need to fix two or three to fix it. I recommend picking one test and focus on fixing that, once it's fixed run the suite again and start over 👍 |
|
@gaborbernat
|
|
@s-vitaliy I found temporal solution. pip install -U virtualenv
python2 -m virtualenv venv
lipo -remove arm64e venv/bin/python -o venv/bin/python
|
|
@isac322 The given workaround dint work for me
|
|
@isac322 Thank you for looking into it. I wanted to let you know that the same problem happens no matter if an M1 or intel-based chip is used. It started to happen specifically after the Monterey upgrade. Also, I'm getting the same error on lipo command. |
|
Hello @gaborbernat, I found the problem but I'm not sure how to solve it. I have a hack fix. I'll share it below.
On sys documentation for python 2.7, it states that this is set on compile-time. So I believe unless any explicit intervention seems like there is no way to set this because we basically copy python compiled files into the virtualenv via global self do. We could have updated it using the custom site.py created for python2 however, that's not loaded due to sys.prefix not pointing the virtualenv directory where custom site.py placed. What I have done is, I created sitecustomizate.py under /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ I don't have time at the moment to continue working on this. There must be a much better way to solve this. If you can guide me, I'd love to create a PR to contribute. Copying custom site.py content under sitecustomize.py not remapping paths also due to reloading global site.py, global site-packages gets added to the PATH even though it becomes the last. There are so many tests failing one main reason is what I describe above another is we are using Tox to run tests which uses virtualenv :) to create virtual environment for tests. Due to this bug, tox actually uses the global space to install dependencies and also virtualenv custom build. That's why I was seeing, global virtualenv module magically got broken or removed. UPDATE: I'm not sure how Monterey upgrade affected this prefix problem. I found a StackOverflow post stating Maybe this has changed, therefore, path manipulation by the site.py not working anymore. |
|
@gaborbernat Sorry to bug you. Because I haven’t figured out why after Monterey upgrade custom site.py is not loading before global python site.py, I had to update site.py of python source code to work smarter like python3 site.py does and be able to read pyvenv.cfg file if present. I'm attaching it here. With this change, no need to use fixed path for macos. So had to change _executables method under |
|
Most of the magic on Python 2 happens here https://github.com/pypa/virtualenv/blob/main/src/virtualenv/create/via_global_ref/builtin/python2/site.py#L10-L24 🤔 so the question is why is this file not doing what you want. |
|
@gaborbernat because it's not getting executed. I tried to manually update site.py and mimic this file functionality into python original site.py but even then it didn't work as expected. Two reasons,
|
|
sys.prefix should be altered by executing this file I've pointed too. By design of how python binary works on V2 it should get executed. If Apple patched/altered that behaviour then all hope is lost and there's no way to get it working. |
|
I found a StackOverflow post stating So I was assuming it'd get executed as you described but it's not. I try with Intel mac, also with an intel mac not upgraded Monterey, it works fine. Only with Monterey, it's not getting executed and sys.prefix points to global Python. Thank you for confirming. |
|
Yeah, you can find more details here https://youtu.be/o1Vue9CWRxU https://gaborbernat.github.io/euro_python_2019/#/ |
|
Thank you these are great. I appreciate. |
|
JFTR, as Apple dropped Python 2 from the base installation few days ago with the 12.3 update, the old workaround (that I used too because I was too lazy to investigate) just disappeared and there's no viable way to run Python 2 on macOS. The problem is not specific to python.org, I've tested it against Homebrew and asdf and they all show the same behavior. I wonder if this is the sign of the universe to drop Python 2 from all my FOSS projects. 🤪 |
Seems like it. |
|
Hopefully someone figures out how to get virtualenv properly working on macOS 12.3. Previously, my virtual environments were working flawlessly on 12.2.1 (intel Mac). After 12.3 removed the native python and I re-installed from python.org, I could not get virtualenvs to work as they did on 12.2.1. As the original poster mentions, python installs new software into the global python2 environment, instead of only in the virtualenv. |
* Fixes a problem in DEVELOPMENT support * Problem seems to be related to macOS 12 Monterey SEE: pypa/virtualenv#2284 * Use special section in "tox.ini" and use "pip install --user ..." to fix the problem for now. * ADDED: "py27" to "envlist" again.
* Update dependencies to newer versions * tox: Cleanup config-file * tox: Fix problem w/ python2.7 - Need to use "pip install --user ..." for now - Scripts seem no longer to be installed HINT: Seems to be related to virtualenv and macOS 12 Monterey SEE: pypa/virtualenv#2284
* Update dependencies to newer versions * tox: Cleanup config-file * tox: Fix problem w/ python2.7 - Need to use "pip install --user ..." for now - Scripts seem no longer to be installed HINT: Seems to be related to virtualenv and macOS 12 Monterey SEE: pypa/virtualenv#2284
* Fixes a problem in DEVELOPMENT support * Problem seems to be related to macOS 12 Monterey SEE: pypa/virtualenv#2284 * Use special section in "tox.ini" and use "pip install --user ..." to fix the problem for now. * ADDED: "py27" to "envlist" again.
* Need to use "pip install --user ..." for now * Scripts seem no longer to be installed HINT: Seems to be related to virtualenv and macOS 12 Monterey SEE: pypa/virtualenv#2284
|
It looks like virtualenv's trick of adding a symlink to I suspect macOS 12.3 affected the behaviour of the following block in getpath.c: https://github.com/python/cpython/blob/v2.7.18/Modules/getpath.c#L445 where As a workaround: Your virtualenv structure would look something like: Keeping the Python dylib name the same and adding the |
Bumps [virtualenv](https://github.com/pypa/virtualenv) from 20.14.0 to 20.14.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/virtualenv/blob/main/docs/changelog.rst">virtualenv's changelog</a>.</em></p> <blockquote> <h2>v20.14.1 (2022-04-11)</h2> <p>Features - 20.14.1</p> <pre><code>- Support for creating a virtual environment from a Python 2.7 framework on macOS 12 - by :user:`nickhutchinson`. (`[#2284](pypa/virtualenv#2284) <https://github.com/pypa/virtualenv/issues/2284>`_) <p>Bugfixes - 20.14.1 </code></pre></p> <ul> <li>Upgrade embedded setuptools to <code>62.1.0</code> from <code>61.0.0</code> - by :user:<code>gaborbernat</code>. (<code>[#2327](pypa/virtualenv#2327) <https://github.com/pypa/virtualenv/issues/2327></code>_)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/virtualenv/commit/17cef12d780e512738b8aed8d02eaee35f50d7cd"><code>17cef12</code></a> release 20.14.1</li> <li><a href="https://github.com/pypa/virtualenv/commit/63e80ff9cb3adb16420d2010c8d43eca9082d1d3"><code>63e80ff</code></a> Support Python 2.7 framework-style distributions on macOS 12 (<a href="https://github-redirect.dependabot.com/pypa/virtualenv/issues/2325">#2325</a>)</li> <li><a href="https://github.com/pypa/virtualenv/commit/8c950c22c1bf350b817eea4919f65d499b8c3c26"><code>8c950c2</code></a> Bump embedded setuptools (<a href="https://github-redirect.dependabot.com/pypa/virtualenv/issues/2324">#2324</a>)</li> <li><a href="https://github.com/pypa/virtualenv/commit/ec2c01e65712966912df763d1727b97ae4ae134a"><code>ec2c01e</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://github-redirect.dependabot.com/pypa/virtualenv/issues/2329">#2329</a>)</li> <li><a href="https://github.com/pypa/virtualenv/commit/1aa302f30b43307556e851617b9ea2ba94d44535"><code>1aa302f</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/virtualenv/issues/2323">#2323</a> from pypa/release-20.14.0</li> <li><a href="https://github.com/pypa/virtualenv/commit/e2822f79cc2667178ac50cde191a858161a39fbf"><code>e2822f7</code></a> Fix changelog</li> <li>See full diff in <a href="https://github.com/pypa/virtualenv/compare/20.14.0...20.14.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
|
Another workaround is to use the Tested with Python.org's 2.7.18 and |
* Update dependencies to newer versions * tox: Cleanup config-file * tox: Fix problem w/ python2.7 - Need to use "pip install --user ..." for now - Scripts seem no longer to be installed HINT: Seems to be related to virtualenv and macOS 12 Monterey SEE: pypa/virtualenv#2284
* Use tox < 4.0 for now (hint: tox 4.x does not support python2) * Use virtualenv < 20.22.0 to retain support for Python 2.7, Python <= 3.6 * tox.ini: Remove py27 section, virtualenv >= 20.14.1 fixes #2284 issue SEE: pypa/virtualenv#2284 * py.requirements/ci.tox.txt: Simplify and use "testing.txt" requirements
* Use tox < 4.0 for now (hint: tox 4.x does not support python2) * Use virtualenv < 20.22.0 to retain support for Python 2.7, Python <= 3.6 * tox.ini: Remove py27 section, virtualenv >= 20.14.1 fixes #2284 issue SEE: pypa/virtualenv#2284 * py.requirements/ci.tox.txt: Simplify and use "testing.txt" requirements
* Update dependencies to newer versions * tox: Cleanup config-file * tox: Fix problem w/ python2.7 - Need to use "pip install --user ..." for now - Scripts seem no longer to be installed HINT: Seems to be related to virtualenv and macOS 12 Monterey SEE: pypa/virtualenv#2284
* Use tox < 4.0 for now (hint: tox 4.x does not support python2) * Use virtualenv < 20.22.0 to retain support for Python 2.7, Python <= 3.6 * tox.ini: Remove py27 section, virtualenv >= 20.14.1 fixes #2284 issue SEE: pypa/virtualenv#2284 * py.requirements/ci.tox.txt: Simplify and use "testing.txt" requirements
* Use tox < 4.0 for now (hint: tox 4.x does not support python2) * Use virtualenv < 20.22.0 to retain support for Python 2.7, Python <= 3.6 * tox.ini: Remove py27 section, virtualenv >= 20.14.1 fixes #2284 issue SEE: pypa/virtualenv#2284 * py.requirements/ci.tox.txt: Simplify and use "testing.txt" requirements
Issue
Hello, we faced an issuie with virtualenv when it is used in python, downloaded form python.org: https://www.python.org/download/releases/2.7/ (Mac Installer disk image)
If virtualenv is created by virtualenv, installed in this package, pip ignores virtualenv and always installs packages into system site-packages:
testbot@mac12-1005 ~ % ./broken_venv/bin/python -m pip install pytest
<cut output>
testbot@mac12-1005 ~ % ./broken_venv/bin/python -m pip show pytest
<cut output>
Location: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
If virtuelenv is created by built-in python (/usr/bin/python), virtualenv works correctly:
testbot@mac12-1005 ~ % ./correct_venv/bin/python -m pip install pytest
<cut output>
testbot@mac12-1005 ~ % ./correct_venv/bin/python -m pip show pytest
<cut output>
Location: /Users/testbot/correct_venv/lib/python2.7/site-packages
For some reason, new site-packages in broken_venv does not contain site-packages of new virtualenv:
Also I noticed that broken virtualenv does not detect base_prefix: "base_prefix": null and correct virtualenv contains "base_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7" (see detailed of bouth commands below)
We use test framework that need to be configured to access mac os UI for UI testing automation, so we cannot use directly /usr/bin/python and need to install pyton via dmg image.
Is it possible to fix this issue or create workaround to make virtualenv work correctly in our case?
Environment
pip listof the host python wherevirtualenvis installed:Output of the virtual environment creation
Incorrect virutualenv creation log
Make sure to run the creation with
-vvv --with-traceback:Correct virtualenv creation log
The text was updated successfully, but these errors were encountered: