RuleBasedStateMachine tests do not work and are not documented

[pspacek]

It seems that tests based on RuleBasedStateMachine are not supported. I've tried to full-text search for "RuleBased" and "stateful" in hypofuzz docs but have not seen any mention this is not supported.

After some digging I've found a mention in the sources:

# Skip state-machine classes, since they're not

but it's a bit mysterious :-)

Maybe collector could print a message when in encounters unsupported case instead of silently ignoring it?

Versions tested

• hypofuzz 24.2.3
• hypothesis 6.100.1
• pytest 8.0.2

Steps to reproduce

1. Copy & paste example code from https://hypothesis.readthedocs.io/en/hypothesis-python-4.57.1/stateful.html into a file, say test_example.py.

2. Run hypothesis fuzz test_example.py

Output

Usage: hypothesis fuzz [OPTIONS] [-- PYTEST_ARGS]
Try 'hypothesis fuzz -h' for help.

Error: No property-based tests were collected

Further details

With a slight modification to hypofuzz/interface.py we can get more verbose output. Diff:

@@ -81,8 +81,8 @@ def _get_hypothesis_tests_with_pytest(args: Iterable[str]) -> List["FuzzProcess"
             ],
             plugins=[collector],
         )
+    print(out.getvalue())  # noqa
     if ret:
-        print(out.getvalue())  # noqa
         print(f"Exiting because pytest returned exit code {ret}")  # noqa
         sys.exit(ret)
     return collector.fuzz_targets

Output:

================================ test session starts =================================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.5.0
rootdir: /tmp/ste/orig
plugins: hypothesis-6.100.1, dash-2.16.1
collected 1 item

<Dir orig>
  <Module test_example.py>
    <UnitTestCase TestDBComparison>
      <TestCaseFunction runTest>
crashed in test_example.py::TestDBComparison::runTest 'function' object has no attribute 'hypothesis'

============================= 1 test collected in 0.02s ==============================

[pspacek]

Additional note, after commenting out print around catch-all except I get more detailed traceback:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 273, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/logging.py", line 783, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/warnings.py", line 118, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1367, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 337, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 810, in perform_collect
INTERNALERROR>     hook.pytest_collection_finish(session=self)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/hypofuzz/interface.py", line 58, in pytest_collection_finish
INTERNALERROR>     fuzz = FuzzProcess.from_hypothesis_test(
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/hypofuzz/hy.py", line 96, in from_hypothesis_test
INTERNALERROR>     given_kwargs=wrapped_test.hypothesis._given_kwargs,
INTERNALERROR>                  ^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> AttributeError: 'function' object has no attribute 'hypothesis'

[Zac-HD]

If someone is going to spend a few hours on this, we should probably add the logic necessary to support RuleBasedStateMachine tests instead of working around that - it's a bit fiddly but not that hard, and we'll definitely want it before HypoFuzz graduates from alpha to beta 🙂

[eivindjahren]

It was also confusing to me why my tests were being skipped, and after debugging the HypoFuzz _ItemCollector I realized it was because the function was silently using the monkeypatch fixture through an autouse fixture. Since the monkeypatch fixture is not autouse, but automatically applied by the autouse fixture, all tests were skipped.