Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pytest-cov fails with pytest-xdist and dynamic_context = "test_function" #604

Closed
masaccio opened this issue Aug 8, 2023 · 2 comments · Fixed by #657
Closed

pytest-cov fails with pytest-xdist and dynamic_context = "test_function" #604

masaccio opened this issue Aug 8, 2023 · 2 comments · Fixed by #657

Comments

@masaccio
Copy link

masaccio commented Aug 8, 2023 •
edited
Loading

Summary

I am trying to use pytest-cov fails with pytest-xdist and want to get coverage for each trace function to ultimately see which lines and arcs are covered by which test function (which I'll then use to create a minimal set of tests that generate n% coverage eliminating redundant tests).

I can use a static context with --cov-context=test together with pytest-xdist but when I try dynamic_context = "test_function" I get an internal error.

Reproducer

This works:

git clone [email protected]:masaccio/numbers-parser.git
poetry install
poetry run pytest -n logical tests/test_version.py

The relevant sections of my pyproject.toml are:

[tool.coverage.run]
branch = true
# dynamic_context = "test_function"
omit = ["src/numbers_parser/generated/*.py"]

[tool.coverage.html]
directory = "coverage_html_report"
show_contexts = true

[tool.pytest.ini_options]
addopts = "--cov=src/numbers_parser --cov-report=term-missing:skip-covered --cov-context=test"

Edit pyproject.toml to remove the comment for dynamic_context. I also removed --cov-context=test but that doesn't make any difference in that the internal error still happens.

[tool.coverage.run]
branch = true
dynamic_context = "test_function"
omit = ["src/numbers_parser/generated/*.py"]

Then re-run poetry run pytest -n logical tests/test_version.py and I observe an internal error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1173, in _execute
INTERNALERROR>     return self.con.execute(sql, parameters)    # type: ignore[arg-type]
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> sqlite3.OperationalError: no such table: file
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1178, in _execute
INTERNALERROR>     return self.con.execute(sql, parameters)    # type: ignore[arg-type]
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> sqlite3.OperationalError: no such table: file
INTERNALERROR> 
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 133, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pytest_cov/plugin.py", line 298, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pytest_cov/engine.py", line 348, in finish
INTERNALERROR>     self.cov.save()
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/control.py", line 757, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>            ^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/control.py", line 838, in get_data
INTERNALERROR>     self._post_save_work()
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/control.py", line 869, in _post_save_work
INTERNALERROR>     self._data.touch_files(paths, plugin_name)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 615, in touch_files
INTERNALERROR>     self._file_id(filename, add=True)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 417, in _file_id
INTERNALERROR>     self._file_map[filename] = con.execute_for_rowid(
INTERNALERROR>                                ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1219, in execute_for_rowid
INTERNALERROR>     with self.execute(sql, parameters) as cur:
INTERNALERROR>   File "/opt/homebrew/Cellar/[email protected]/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
INTERNALERROR>     return next(self.gen)
INTERNALERROR>            ^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1207, in execute
INTERNALERROR>     cur = self._execute(sql, parameters)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1195, in _execute
INTERNALERROR>     raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
INTERNALERROR> coverage.exceptions.DataError: Couldn't use data file '/Users/jon/Downloads/numbers-parser/.coverage.Jons-MacBook-Air.93985.543340': no such table: file

My config includes some adopts for pytest but commenting those out and using poetry run pytest -n logical --cov=src/numbers_parser tests/test_version.py alone is sufficient to crash. Playing around with arguments I can see ordering doesn't help but the number of threads does. At least on my test machine:

  • poetry run pytest --cov=src/numbers_parser -n 1 tests/test_version.py succeeds
  • poetry run pytest --cov=src/numbers_parser -n 2 tests/test_version.py succeeds
  • poetry run pytest --cov=src/numbers_parser -n 3 tests/test_version.py fails

Versions

% poetry run pip freeze | grep pytest
pytest==7.4.0
pytest-check==1.3.0
pytest-console-scripts==1.4.1
pytest-cov==4.1.0
pytest-profiling==1.7.0
pytest-xdist==3.3.1
% poetry run python --version
Python 3.11.4
ionelmc added a commit that referenced this issue Sep 17, 2024
@ionelmc
Add reproducer for weird xdist dynamic_context interaction. Ref #604.
6fa6109
@ionelmc
Copy link
Member

ionelmc commented Sep 18, 2024

I have managed to reproduce the problem but I don't understand what's the point of using dynamic_context=test_function when pytest-cov's --cov-context option already handles that for you.

The problem is caused by should_start_context_test_function triggering a switch in the middle of some xdist internals (xdist.scheduler.load.tests_finished).

Please justify your usecase.

I think the only reasonable thing to do here is make pytest-cov raise an error when it detects that you are using dynamic_context=test_function and xdist, and a warning when you are using just dynamic_context=test_function.

@ionelmc ionelmc mentioned this issue Sep 18, 2024
Merged
@ionelmc
Copy link
Member

ionelmc commented Sep 18, 2024

To give some context, this is the implementation of should_start_context_test_function (the context switcher set if you have dynamic_context=test_function):

def should_start_context_test_function(frame: FrameType) -> str | None:
    """Is this frame calling a test_* function?"""
    co_name = frame.f_code.co_name
    if co_name.startswith("test") or co_name == "runTest":
        return qualname_from_frame(frame)
    return None

You will notice that xdist.scheduler.load.tests_finished will unfortunately match, and trigger a switch at a very inopportune time:

79527.ff6f: Erasing data file '/tmp/pytest-of-ionel/pytest-75/test_dynamic_context0/.coverage.dev.79527.XYVLYICx.master1'
79527.ff6f:   File "<frozen runpy>", line 198, in _run_module_as_main
79527.ff6f:   File "<frozen runpy>", line 88, in _run_code
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pytest/__main__.py", line 7, in <module>
79527.ff6f:     raise SystemExit(pytest.console_main())
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197, in console_main
79527.ff6f:     code = main()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174, in main
79527.ff6f:     ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
79527.ff6f:     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
79527.ff6f:     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
79527.ff6f:     res = hook_impl.function(*args)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/main.py", line 332, in pytest_cmdline_main
79527.ff6f:     return wrap_session(config, _main)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/main.py", line 285, in wrap_session
79527.ff6f:     session.exitstatus = doit(config, session) or 0
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/main.py", line 339, in _main
79527.ff6f:     config.hook.pytest_runtestloop(session=session)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
79527.ff6f:     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
79527.ff6f:     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
79527.ff6f:     res = hook_impl.function(*args)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
79527.ff6f:     self.loop_once()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/xdist/dsession.py", line 149, in loop_once
79527.ff6f:     if self.sched.tests_finished:
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/xdist/scheduler/load.py", line 84, in tests_finished
79527.ff6f:     @property
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/collector.py", line 404, in switch_context
79527.ff6f:     self.flush_data()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/collector.py", line 486, in flush_data
79527.ff6f:     self.covdata.add_arcs(self.mapped_file_dict(arc_data))
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 123, in _wrapped
79527.ff6f:     return method(self, *args, **kwargs)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 531, in add_arcs
79527.ff6f:     self._start_using()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 860, in _start_using
79527.ff6f:     self.erase()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 831, in erase
79527.ff6f:     self._debug.write(f"Erasing data file {self._filename!r}")
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/debug.py", line 107, in write
79527.ff6f:     dump_stack_frames(out=self.output, skip=1)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/debug.py", line 268, in dump_stack_frames
79527.ff6f:     ''.join(traceback.format_stack()),

ionelmc added a commit that referenced this issue Sep 23, 2024
@ionelmc
Add reproducer for weird xdist dynamic_context interaction. Ref #604.
c87e546
@shaypal5 shaypal5 mentioned this issue Nov 2, 2024
Merged
@ivantodorovich ivantodorovich mentioned this issue Nov 4, 2024
Merged
moylop260 added a commit to vauxoo-dev/pre-commit-vauxoo that referenced this issue Nov 4, 2024
@moylop260
[REF] tox.ini: Add compatibility with new pyttest
ef62e9a 
Related to code https://github.com/pytest-dev/pytest-cov/blob/95404375a0e436178e012e20b8865c23c54c8a50/src/pytest_cov/engine.py\#L251C25-L251C26

Related to issue pytest-dev/pytest-cov#604
moylop260 added a commit to Vauxoo/pre-commit-vauxoo that referenced this issue Nov 4, 2024
@moylop260
[REF] tox.ini: Add compatibility with new pyttest
9f5e60f 
Related to code https://github.com/pytest-dev/pytest-cov/blob/95404375a0e436178e012e20b8865c23c54c8a50/src/pytest_cov/engine.py\#L251C25-L251C26

Related to issue pytest-dev/pytest-cov#604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle issue 604 pytest-dev/pytest-cov
2 participants
@ionelmc @masaccio