Prevent sys.prefix from leaking into child process on macOS

[cjolowicz]

Fixes #1643

This PR fixes a bug on macOS where virtualenv, when run inside a virtual environment, identifies every Python interpreter as the interpreter used to generate the virtual environment.

There is a related upstream bug at bpo22490, with an open PR at python/cpython#9516.

Setup

• Your system is macOS.
• You have not run virtualenv on this system before, or you have removed the user data directory at ~/Library/Application\ Support/virtualenv/.
• You have a framework build of Python on your PATH. Example: /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (via Homebrew)
  (This is usually accessed as /usr/local/opt/python/bin/python3.7.)
• You have another version of Python on your PATH. Example: ~/.pyenv/versions/3.8.1/bin/python3.8 (via pyenv)

Repro

1. Create a virtual environment using the framework build:

   /usr/local/opt/python/bin/python3.7 -m venv /tmp/venv

2. Install virtualenv into the virtual environment.

   /tmp/venv/bin/python -m pip install virtualenv

3. Create a virtual environment for 3.8 using virtualenv.

   /tmp/venv/bin/virtualenv -p python3.8 /tmp/venv-3.8

Step 3 fails with the following error message:

RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8'

Analysis

Here is a step-by-step analysis.

1. User invokes /tmp/venv/bin/virtualenv -p python3.8 /tmp/venv-3.8.
2. OS executes /tmp/venv/bin/python, a symlink to the framework build of Python 3.7.
3. Framework build executes the actual Python binary, passing the original interpreter path (/tmp/venv/bin/python) in the environment variable __PYVENV_LAUNCHER__.
4. Virtualenv spawns a subprocess executing ~/.pyenv/shims/python3.8 /tmp/venv/lib/.../virtualenv/discovery/py_info.py to obtain information about the interpreter.
5. The shim executes ~/.pyenv/versions/3.8.1/bin/python3.8 (via ~/.pyenv/libexec/pyenv-exec).
6. The interpreter imports ~/.pyenv/versions/3.8.1/lib/python3.8/site.py during initialization.
7. The site.py script sees that the platform is darwin and the __PYVENV_LAUNCHER__ environment variable is set. It sets sys._base_executable to the value of __PYVENV_LAUNCHER__, and sys.prefix to the grand-parent directory of this path. sys.prefix is now /tmp/venv, while sys.base_prefix still points to the previous value of sys.prefix, ~/.pyenv/versions/3.8.1.
8. Virtualenv sees that sys.prefix and sys.base_prefix differ, and assumes that the interpreter is running in a virtual environment. It determines the system executable using sys._base_executable, /tmp/venv/bin/python.
9. Virtualenv uses /tmp/venv/bin/python to identify the interpreter, overriding the information previously gathered from the pyenv-provided Python 3.8. The interpreter is identified as Python 3.7.

References to source code:

Step 7:

https://github.com/python/cpython/blob/d4d17fd2cf69e7c8f4cd03fbf2d575370945b952/Lib/site.py#L460-L461

Step 8:

virtualenv/src/virtualenv/discovery/py_info.py

Lines 94 to 107 in 5b88149

	def _fast_get_system_executable(self):
	"""Try to get the system executable by just looking at properties"""
	if self.real_prefix or (
	self.base_prefix is not None and self.base_prefix != self.prefix
	): # if this is a virtual environment
	if self.real_prefix is None:
	base_executable = getattr(sys, "_base_executable", None) # some platforms may set this to help us
	if base_executable is not None: # use the saved system executable if present
	if sys.executable != base_executable: # we know we're in a virtual environment, cannot be us
	return base_executable
	return None # in this case we just can't tell easily without poking around FS and calling them, bail
	# if we're not in a virtual environment, this is already a system python, so return the original executable
	# note we must choose the original and not the pure executable as shim scripts might throw us off
	return self.original_executable

Step 9:

virtualenv/src/virtualenv/discovery/py_info.py

Lines 284 to 294 in 5b88149

	@classmethod
	def from_exe(cls, exe, raise_on_error=True, ignore_cache=False, resolve_to_host=True):
	"""Given a path to an executable get the python information"""
	# this method is not used by itself, so here and called functions can import stuff locally
	from virtualenv.discovery.cached_py_info import from_exe
	proposed = from_exe(cls, exe, raise_on_error=raise_on_error, ignore_cache=ignore_cache)
	# noinspection PyProtectedMember
	if isinstance(proposed, PythonInfo) and resolve_to_host:
	proposed = proposed._resolve_to_system(proposed)
	return proposed

virtualenv/src/virtualenv/discovery/py_info.py

Lines 309 to 326 in 5b88149

	@classmethod
	def _resolve_to_system(cls, target):
	start_executable = target.executable
	prefixes = OrderedDict()
	while target.system_executable is None:
	prefix = target.real_prefix or target.base_prefix or target.prefix
	if prefix in prefixes:
	for at, (p, t) in enumerate(prefixes.items(), start=1):
	logging.error("%d: prefix=%s, info=%r", at, p, t)
	logging.error("%d: prefix=%s, info=%r", len(prefixes) + 1, prefix, target)
	raise RuntimeError("prefixes are causing a circle {}".format("|".join(prefixes.keys())))
	prefixes[prefix] = target
	target = target.discover_exe(prefix=prefix, exact=False)
	if target.executable != target.system_executable:
	target = cls.from_exe(target.system_executable)
	target.executable = start_executable
	return target

---

Thanks for contributing a pull request, see checklist all is good!

• wrote descriptive pull request text
• added/updated test(s)
• updated/extended the documentation
• added news fragment in docs/changelog folder

[cjolowicz]

Thanks for merging! ...and sorry, did not get around to writing test and changelog earlier.

Updated PR description with repro and analysis.