Can't open orphan path on bare importlib.resources.files() under zipapp

[winstontsai]

Bug report

Bug description:

Context: In Python 3.12 the function importlib.resources.files() was updated: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

Changed in version 3.12: package parameter was renamed to anchor. anchor can now be a non-package module and if omitted will default to the caller’s module. package is still accepted for compatibility but will raise a DeprecationWarning. Consider passing the anchor positionally or using importlib_resources >= 5.10 for a compatible interface on older Pythons.

The Issue: When using importlib.resources.files(), I am able to open and read a text file included as package data in my package. However, if I zip up my package with zipapp and run the .pyz file, it is unable to read the text file.
If I use importlib.resources.files("mypkg") it will work in both cases.

Minimal example:

Create a project directory named mypkg with the src layout.

Create the following files.

mypkg/src/mypkg/a.py

import importlib.resources

def main():
    x = importlib.resources.files() / "data.txt"
    print(x.read_text())

mypkg/src/mypkg/data.txt

datadata

In pyproject.toml make sure to set up the entry script and include the data file.

mypkg/pyproject.toml

[project.scripts]
myscript = "mypkg.a:main"

[tool.setuptools.package-data]
mypkg = ["*.txt"]

Install the package into a fresh Python (virtual) environment with pip.
Run myscript and verify it prints datadata.

Now back in the mypkg project root folder, we will generate a .pyz file with zipapp and run the program from the .pyz file.

Run pip install . --target buildtemp --upgrade && python3 -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env python3' -o mypkg.pyz in the terminal.

Run ./mypkg.pyz. This results in an error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/win/code/sandbox/python/mypkg/./mypkg.pyz/__main__.py", line 3, in <module>
  File "/home/win/code/sandbox/python/mypkg/./mypkg.pyz/mypkg/a.py", line 6, in main
  File "/home/win/.local/share/mise/installs/python/3.12.4/lib/python3.12/importlib/resources/abc.py", line 89, in read_text
    with self.open(encoding=encoding) as strm:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/win/.local/share/mise/installs/python/3.12.4/lib/python3.12/importlib/resources/_adapters.py", line 139, in open
    raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path

Note that if you change x = importlib.resources.files() / "data.txt" to x = importlib.resources.files("mypkg") / "data.txt", it will work when running the .pyz file.

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Linked PRs

• gh-121735: Fix module-adjacent references in zip files #123037
• [3.13] gh-121735: Fix module-adjacent references in zip files (GH-123037) #123986
• [3.12] gh-121735: Fix module-adjacent references in zip files (gh-123037) #124011

[wimglenn]

ping @jaraco in case you missed this one

[jaraco]

The code that handles the module inference is here:

cpython/Lib/importlib/resources/_common.py

Lines 75 to 104 in 6ae942f

	@functools.singledispatch
	def resolve(cand: Optional[Anchor]) -> types.ModuleType:
	return cast(types.ModuleType, cand)
	@resolve.register
	def _(cand: str) -> types.ModuleType:
	return importlib.import_module(cand)
	@resolve.register
	def _(cand: None) -> types.ModuleType:
	return resolve(_infer_caller().f_globals['__name__'])
	def _infer_caller():
	"""
	Walk the stack and find the frame of the first caller not in this module.
	"""
	def is_this_file(frame_info):
	return frame_info.filename == __file__
	def is_wrapper(frame_info):
	return frame_info.function == 'wrapper'
	not_this_file = itertools.filterfalse(is_this_file, inspect.stack())
	# also exclude 'wrapper' due to singledispatch in the call stack
	callers = itertools.filterfalse(is_wrapper, not_this_file)
	return next(callers).frame

Next step is to determine what expectation is missed when running under zipapp.

[jaraco]

I was able to replicate the issue without going through the wheel build process:

 draft 🐚 mkdir -p buildtemp/mypkg
 draft 🐚 cat > buildtemp/mypkg/a.py
import importlib.resources


def main():
    x = importlib.resources.files() / "data.txt"
    print(x.read_text())
 draft 🐚 echo datadata > buildtemp/mypkg/data.txt
 draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz
 draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module>
    mypkg.a.main()
    ~~~~~~~~~~~~^^
  File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main
    print(x.read_text())
          ~~~~~~~~~~~^^
  File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text
    with self.open(encoding=encoding) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open
    raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path

[jaraco]

The issue is that CompatibilityFiles is being activated for loading resources.

 draft 🐚 py -m pdb mypkg.pyz
> /Users/jaraco/draft/mypkg.pyz/__main__.py(2)<module>()
-> import mypkg.a
(Pdb) c
Traceback (most recent call last):
  File "/opt/python/lib/python3.13/pdb.py", line 2435, in main
    pdb._run(target)
    ~~~~~~~~^^^^^^^^
  File "/opt/python/lib/python3.13/pdb.py", line 2192, in _run
    self.run(target.code)
    ~~~~~~~~^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/bdb.py", line 651, in run
    exec(cmd, globals, locals)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/draft/mypkg.pyz/__main__.py", line 3, in <module>
    mypkg.a.main()
    ~~~~~~~~~~~~^^
  File "/Users/jaraco/draft/mypkg.pyz/mypkg/a.py", line 6, in main
    print(x.read_text())
          ~~~~~~~~~~~^^
  File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text
    with self.open(encoding=encoding) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open
    raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /opt/python/lib/python3.13/importlib/resources/_adapters.py(139)open()
-> raise FileNotFoundError("Can't open orphan path")
(Pdb) u
> /opt/python/lib/python3.13/importlib/resources/abc.py(89)read_text()
-> with self.open(encoding=encoding) as strm:
(Pdb) u
> /Users/jaraco/draft/mypkg.pyz/mypkg/a.py(6)main()
-> print(x.read_text())
(Pdb) x
<importlib.resources._adapters.CompatibilityFiles.OrphanPath object at 0x41f4c584f90>

CompatibilityFiles gets employed when there's no proper resource provider found. I'll figure out why.

[jaraco]

It's clear that the bare call to files() is getting CompatibiltyFiles when the call with an anchor returns a proper MultiplexedPath:

(Pdb) importlib.resources.files()
<importlib.resources._adapters.CompatibilityFiles.SpecPath object at 0x41f4c595a10>
(Pdb) importlib.resources.files('mypkg')
MultiplexedPath('/Users/jaraco/draft/mypkg.pyz/mypkg/')

By changing the main() function to emit the inferred caller's name,

def main():
    print(importlib.resources._common._infer_caller().f_globals['__name__'])

I can confirm that the inferred name is correct when it emits mypkg.a.

Interestingly, if I then replace the main function with a static 'mypkg.a', the reported error occurs.

def main():
    x = importlib.resources.files('mypkg.a') / "data.txt"
    print(x.read_text())

So the issue seems to be that files('mypkg') works as expected but files('mypkg.a') does not, and it's only coincidental that files() calls files('mypkg.a').

importlib.resources has a behavior that when given a module will resolve resources in the package of that module.

It seems that the zipimporter as attached to the module is not resolving a resource reader:

(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg.a')
(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg')
(Pdb) importlib.import_module('mypkg').__spec__.loader.get_resource_reader('mypkg')
<importlib.resources.readers.NamespaceReader object at 0x3df2e585410>

I'm slightly surprised to see NamespaceReader and not a ZipReader.

[jaraco]

It seems the issue traces to the logic in get_resource_reader:

cpython/Lib/zipimport.py

Lines 258 to 270 in 6ae942f

	def get_resource_reader(self, fullname):
	"""Return the ResourceReader for a package in a zip file.
	If 'fullname' is a package within the zip file, return the
	'ResourceReader' object for the package. Otherwise return None.
	"""
	try:
	if not self.is_package(fullname):
	return None
	except ZipImportError:
	return None
	from importlib.readers import ZipReader
	return ZipReader(self, fullname)

When checking loader.is_package for mypkg.a, it returns False:

(Pdb) loader = importlib.import_module('mypkg.a').__spec__.loader
(Pdb) loader.is_package('mypkg.a')
False

In the early days of importlib.resources, all resources needed to be relative to a package (and not a module). Later, that restriction was relaxed to allow resources to be loaded relative to a module.

But it seems that get_resource_reader didn't get the corresponding treatment.

I've confirmed the same issue occurs when using importlib_resources:

 draft [1] 🐚 cat buildtemp/mypkg/a.py
import importlib_resources


def main():
    x = importlib_resources.files('mypkg.a') / "data.txt"
    print(x.read_text())
 draft 🐚 pip install -t buildtemp importlib_resources
Collecting importlib_resources
  Using cached importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB)
Using cached importlib_resources-6.4.0-py3-none-any.whl (38 kB)
Installing collected packages: importlib_resources
Successfully installed importlib_resources-6.4.0
 draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz
 draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
...
FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txt

I suspect the resolution is going to be update get_resource_reader to accept a module.

But first things first, let's figure out why this use-case isn't covered by the tests.

[jaraco]

As a check, if I remove the is_package check:

 cpython main 🐚 git diff
diff --git a/Lib/zipimport.py b/Lib/zipimport.py
index f2724dd0268..c0a4abcd95d 100644
--- a/Lib/zipimport.py
+++ b/Lib/zipimport.py
@@ -256,16 +256,8 @@ def load_module(self, fullname):
 
 
     def get_resource_reader(self, fullname):
-        """Return the ResourceReader for a package in a zip file.
-
-        If 'fullname' is a package within the zip file, return the
-        'ResourceReader' object for the package.  Otherwise return None.
+        """Return the ResourceReader for a module in a zip file.
         """
-        try:
-            if not self.is_package(fullname):
-                return None
-        except ZipImportError:
-            return None
         from importlib.readers import ZipReader
         return ZipReader(self, fullname)

The routine then fails, but with a different error:

 draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module>
    mypkg.a.main()
    ~~~~~~~~~~~~^^
  File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main
    print(x.read_text())
          ~~~~~~~~~~~^^
  File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 426, in read_text
    with self.open('r', encoding, *args, **kwargs) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 391, in open
    raise FileNotFoundError(self)
FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txt

That's encouraging, suggesting that the ZipReader is getting involved, but that it still doesn't honor resources adjacent to modules.

[jaraco]

I started working on a test, seeking to extend these tests to zip files, but realized that the current test structure isn't easily amenable to extension to zip. In python/importlib_resources@3faf336, I've done some refactoring in order to leverage the zip fixtures for this purpose.

[jaraco]

I'm releasing importlib_resources 6.4.1 with a fix, but that fix obviously only applies to the backport. We'll need to apply those changes to CPython as well, likely with the aforementioned diff to zipimport.

[wimglenn]

Quick turnaround, thank you. If I understand devguide correctly, miss-islington should presumably backport the CPython change into 3.12.x as bugfix (but not any earlier versions)?

[jaraco]

That's right. And I think it was 3.12 that introduced this feature, so there shouldn't be any users of earlier versions expecting anything different.

[jaraco]

I've nearly got the feature ported, but there's a failure in one of the new tests (ImplicitContextFileZipTests.test_implicit_files_submodule):

test test_importlib failed -- Traceback (most recent call last):
  File "/Users/jaraco/code/python/cpython/Lib/test/test_importlib/resources/test_files.py", line 123, in test_implicit_files_submodule
    assert importlib.import_module('somepkg.submod').val == 'resources are the best'
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/code/python/cpython/Lib/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1386, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1359, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1330, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 752, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/tmpjcv3yk54/zipped modules.zip/somepkg/submod.py", line 3, in <module>
    val = res.files().joinpath('res.txt').read_text(encoding='utf-8')
  File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 426, in read_text
    with self.open('r', encoding, *args, **kwargs) as strm:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 391, in open
    raise FileNotFoundError(self)
FileNotFoundError: /private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/tmpjcv3yk54/zipped modules.zip/somepkg/submod/res.txt

Somehow the submod is being included in the path (even though that doesn't happen in importlib_resources.

[jaraco]

But when I inspect the reader directly, everything looks fine:

> /Users/jaraco/code/python/cpython/Lib/test/test_importlib/resources/test_files.py(123)test_implicit_files_submodule()
-> breakpoint()
(Pdb) import somepkg.submod
(Pdb) somepkg.submod.__spec__.loader
<zipimporter object "/private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/tmpxzp59s01/zipped modules.zip/somepkg/">
(Pdb) somepkg.submod.__spec__.loader.get_resource_reader('somepkg.submod')
<importlib.resources.readers.ZipReader object at 0x10443c1a0>
(Pdb) reader = somepkg.submod.__spec__.loader.get_resource_reader('somepkg.submod')
(Pdb) reader.prefix
'somepkg/'
(Pdb) reader.files()
Path('/private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/tmpxzp59s01/zipped modules.zip', 'somepkg/')
(Pdb) from importlib.resources._adapters import wrap_spec
(Pdb) spec = wrap_spec(somepkg.submod)
(Pdb) spec
<importlib.resources._adapters.SpecLoaderAdapter object at 0x10443c1a0>
(Pdb) spec.loader.get_resource_reader(spec.name)
<importlib.resources.readers.ZipReader object at 0x104382210>
(Pdb) reader = spec.loader.get_resource_reader(spec.name)
(Pdb) reader.prefix
'somepkg/'
(Pdb) reader.files()
Path('/private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/tmpxzp59s01/zipped modules.zip', 'somepkg/')
(Pdb) reader.files().joinpath('res.txt').read_text()
'resources are the best'

So I'm stumped as to why the same call fails from somepkg.submod.

[jaraco]

I think I found the issue - a lingering importlib_resources reference.

[jaraco]

Yes, that's all it was (whew!).