Why no output sections for Xpass/Xfail? #9834

jeffwright13 · 2022-04-03T19:31:00Z

jeffwright13
Apr 3, 2022

We have separate output sections for FAILURES, PASSES (if the rA option is selected), warnings summary and ERRORS. Why then no output sections for XPASS and XFAIL? Or SKIPPED, for that matter? They show up as separate items in the final summary result count (the last line of the terminal output); Xpass tests do not show in the PASSES section and Xfails do not show in the FAILURES section. Seems like we're missing something here.

Answered by okken

Feb 8, 2024

pytest 8.0.0 reports ouptut for xpass and xfail if you use -rxX or -ra (which includes -rx for xfail and -rX for xpass )

View full answer

nicoddemus · 2022-04-04T12:40:53Z

nicoddemus
Apr 4, 2022
Maintainer

Hi @jeffwright13,

Can you paste what we have today and what you would like to see, to make sure we are all on the same page? Thanks.

3 replies

jeffwright13 Apr 4, 2022
Author

@nicoddemus - please see below for output generated by Pytest 7.1.1 with the -v flag set on command line, and addopts = -rA -v set in pytest.ini. The testscases were created to exercise all the available output options (although currently log_cli is disabled).

As you can see, there are individual output sections (separated by === <SECTION-NAME> === lines) for:

test session starts
ERRORS
FAILURES
warning summary
PASSES
short test summary info
final line with summary statistics (11 failed, 8 passed, 3 skipped, 3 xfailed, 3 xpassed, 22 warnings, 4 errors in 3.20s)

However, there are no output setctions for Xpass, Xfail, or Skipped. In my opinion, because the XFail and XPass results do not get counted in the final Failed and Passed statistics, they should have their own output section. Same would also apply to Skipped.

I hope this makes sense?

nicoddemus Apr 4, 2022
Maintainer

Thanks @jeffwright13!

In my opinion, because the XFail and XPass results do not get counted in the final Failed and Passed statistics,

But they are counted, no?

11 failed, 8 passed, 3 skipped, 3 xfailed, 3 xpassed, 22 warnings, 4 errors in 3.20s

jeffwright13 Apr 4, 2022
Author

Yes, they are counted, so the final statistics are correct. I'm just curious why they don't get their own sections in the terminal output.

And while we are talking about it, how come the "warnings summery" and "test session starts" sections are lower case, but ERRORS, PASSES and FAILS are upper case?

As you can see, I have a preference for consistency. :-)

jeffwright13 · 2022-04-04T13:29:07Z

jeffwright13
Apr 4, 2022
Author

$ pytest -v
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0 -- /Users/jwr003/coding/pytest-fold/venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/jwr003/coding/pytest-fold, configfile: pytest.ini, testpaths: tests
plugins: Faker-13.3.2, fold-0.7.6
collected 32 items

tests/test_pytest_fold_1.py::test_a_ok PASSED                                                                                                                                      [  3%]
tests/test_pytest_fold_1.py::test_b_fail FAILED                                                                                                                                    [  6%]
tests/test_pytest_fold_1.py::test_c_error ERROR                                                                                                                                    [  9%]
tests/test_pytest_fold_1.py::test_d_skip SKIPPED (skipping this test)                                                                                                              [ 12%]
tests/test_pytest_fold_1.py::test_e_xfail XFAIL (xfailing this test)                                                                                                               [ 15%]
tests/test_pytest_fold_1.py::test_f_xpass XPASS (always xfail)                                                                                                                     [ 18%]
tests/test_pytest_fold_1.py::test_g_eval_parameterized[3+5-8] PASSED                                                                                                               [ 21%]
tests/test_pytest_fold_1.py::test_g_eval_parameterized[2+4-6] PASSED                                                                                                               [ 25%]
tests/test_pytest_fold_1.py::test_g_eval_parameterized[6*9-42] FAILED                                                                                                              [ 28%]
tests/test_pytest_fold_1.py::test_1_passes_and_has_logging_output PASSED                                                                                                           [ 31%]
tests/test_pytest_fold_1.py::test_2_fails_and_has_logging_output FAILED                                                                                                            [ 34%]
tests/test_pytest_fold_1.py::test_3_fails FAILED                                                                                                                                   [ 37%]
tests/test_pytest_fold_1.py::test_4_passes PASSED                                                                                                                                  [ 40%]
tests/test_pytest_fold_1.py::test_5_marked_SKIP SKIPPED (unconditional skip)                                                                                                       [ 43%]
tests/test_pytest_fold_1.py::test_6_marked_xfail_but_passes XPASS                                                                                                                  [ 46%]
tests/test_pytest_fold_1.py::test_7_marked_xfail_and_fails XFAIL                                                                                                                   [ 50%]
tests/test_pytest_fold_1.py::test_8_causes_a_warning FAILED                                                                                                                        [ 53%]
tests/test_pytest_fold_1.py::test_9_lorem_fails FAILED                                                                                                                             [ 56%]
tests/test_pytest_fold_1.py::test_10_fail_capturing FAIL stdout not captured, going directly to sys.stdout
FAIL stderr not captured, going directly to sys.stderr
FAILED                                                                                                                         [ 59%]
tests/test_pytest_fold_1.py::test_11_pass_capturing FAILED                                                                                                                         [ 62%]
tests/test_pytest_fold_1.py::test_12_fails_and_has_stdout FAILED                                                                                                                   [ 65%]
tests/test_pytest_fold_1.py::test_13_passes_and_has_stdout PASSED                                                                                                                  [ 68%]
tests/test_pytest_fold_1.py::test_14_causes_error_pass_stderr_stdout_stdlog ERROR                                                                                                  [ 71%]
tests/test_pytest_fold_1.py::test_15_causes_error_fail_stderr_stdout_stdlog ERROR                                                                                                  [ 75%]
tests/test_pytest_fold_2.py::test_i_ok PASSED                                                                                                                                      [ 78%]
tests/test_pytest_fold_2.py::test_ii_fail FAILED                                                                                                                                   [ 81%]
tests/test_pytest_fold_2.py::test_iii_error ERROR                                                                                                                                  [ 84%]
tests/test_pytest_fold_2.py::test_iv_skip SKIPPED (skipping this test)                                                                                                             [ 87%]
tests/test_pytest_fold_2.py::test_v_xfail XFAIL (xfailing this test)                                                                                                               [ 90%]
tests/test_pytest_fold_2.py::test_vi_xpass XPASS (always xfail)                                                                                                                    [ 93%]
tests/test_pytest_fold_warning.py::test_one_warning_fails FAILED                                                                                                                   [ 96%]
tests/test_pytest_fold_warning.py::test_two_warning_passes PASSED                                                                                                                  [100%]

========================================================================================= ERRORS =========================================================================================
_____________________________________________________________________________ ERROR at setup of test_c_error _____________________________________________________________________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

tests/test_pytest_fold_1.py:18: AssertionError
____________________________________________________________ ERROR at setup of test_14_causes_error_pass_stderr_stdout_stdlog ____________________________________________________________
file /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_1.py, line 171
  def test_14_causes_error_pass_stderr_stdout_stdlog(fixture_for_fun):
E       fixture 'fixture_for_fun' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, error_fixture, faker, log_testname, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_1.py:171
____________________________________________________________ ERROR at setup of test_15_causes_error_fail_stderr_stdout_stdlog ____________________________________________________________
file /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_1.py, line 178
  def test_15_causes_error_fail_stderr_stdout_stdlog(fixture_for_fun):
E       fixture 'fixture_for_fun' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, error_fixture, faker, log_testname, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_1.py:178
____________________________________________________________________________ ERROR at setup of test_iii_error ____________________________________________________________________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

tests/test_pytest_fold_2.py:18: AssertionError
======================================================================================== FAILURES ========================================================================================
______________________________________________________________________________________ test_b_fail _______________________________________________________________________________________

    def test_b_fail():
>       assert 0
E       assert 0

tests/test_pytest_fold_1.py:26: AssertionError
___________________________________________________________________________ test_g_eval_parameterized[6*9-42] ____________________________________________________________________________

test_input = '6*9', expected = 42

    @pytest.mark.parametrize("test_input, expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
    def test_g_eval_parameterized(test_input, expected):
>       assert eval(test_input) == expected
E       AssertionError: assert 54 == 42
E        +  where 54 = eval('6*9')

tests/test_pytest_fold_1.py:48: AssertionError
__________________________________________________________________________ test_2_fails_and_has_logging_output ___________________________________________________________________________

log_testname = None

    def test_2_fails_and_has_logging_output(log_testname):
        logger.critical(fake_data())
        logger.error(fake_data())
        logger.warning(fake_data())
        logger.info(fake_data())
        logger.debug(fake_data())
>       assert 0 == 1
E       assert 0 == 1

tests/test_pytest_fold_1.py:77: AssertionError
--------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------
Running test tests.test_pytest_fold_1...
Running test tests.test_pytest_fold_1...
Setting test up...
Setting test up...
Executing test...
Executing test...
Tearing test down...
Tearing test down...
----------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------
INFO     root:test_pytest_fold_1.py:52 Running test tests.test_pytest_fold_1...
INFO     root:test_pytest_fold_1.py:53 Setting test up...
INFO     root:test_pytest_fold_1.py:54 Executing test...
INFO     root:test_pytest_fold_1.py:56 Tearing test down...
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
Chair beautiful build speech close about drop reach.
Chair beautiful build speech close about drop reach.
Activity writer remember home.
Activity writer remember home.
He prove reality. Both knowledge himself change.
He prove reality. Both knowledge himself change.
Report view drop down.
Run painting before event start sing old.
Report view drop down.
Run painting before event start sing old.
Whom test street property list hospital response others.
Whom test street property list hospital response others.
----------------------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------------------
CRITICAL root:test_pytest_fold_1.py:72 Chair beautiful build speech close about drop reach.
ERROR    root:test_pytest_fold_1.py:73 Activity writer remember home.
WARNING  root:test_pytest_fold_1.py:74 He prove reality. Both knowledge himself change.
INFO     root:test_pytest_fold_1.py:75 Report view drop down.
Run painting before event start sing old.
DEBUG    root:test_pytest_fold_1.py:76 Whom test street property list hospital response others.
______________________________________________________________________________________ test_3_fails ______________________________________________________________________________________

log_testname = None

    def test_3_fails(log_testname):
>       assert 0
E       assert 0

tests/test_pytest_fold_1.py:81: AssertionError
--------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------
Running test tests.test_pytest_fold_1...
Running test tests.test_pytest_fold_1...
Setting test up...
Setting test up...
Executing test...
Executing test...
Tearing test down...
Tearing test down...
----------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------
INFO     root:test_pytest_fold_1.py:52 Running test tests.test_pytest_fold_1...
INFO     root:test_pytest_fold_1.py:53 Setting test up...
INFO     root:test_pytest_fold_1.py:54 Executing test...
INFO     root:test_pytest_fold_1.py:56 Tearing test down...
________________________________________________________________________________ test_8_causes_a_warning _________________________________________________________________________________

log_testname = None

    def test_8_causes_a_warning(log_testname):
>       assert api_v1() == 1
E       TypeError: api_v1() missing 1 required positional argument: 'log_testname'

tests/test_pytest_fold_1.py:110: TypeError
--------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------
Running test tests.test_pytest_fold_1...
Running test tests.test_pytest_fold_1...
Setting test up...
Setting test up...
Executing test...
Executing test...
Tearing test down...
Tearing test down...
----------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------
INFO     root:test_pytest_fold_1.py:52 Running test tests.test_pytest_fold_1...
INFO     root:test_pytest_fold_1.py:53 Setting test up...
INFO     root:test_pytest_fold_1.py:54 Executing test...
INFO     root:test_pytest_fold_1.py:56 Tearing test down...
___________________________________________________________________________________ test_9_lorem_fails ___________________________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x10cce3bb0>

    def test_9_lorem_fails(capsys):
        lorem = """"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

        Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?

        At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat."""
        print(lorem)
>       assert False
E       assert False

tests/test_pytest_fold_1.py:122: AssertionError
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

    Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?

    At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.
_________________________________________________________________________________ test_10_fail_capturing _________________________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x10cdb2dd0>

    def test_10_fail_capturing(capsys):
        print("FAIL this stdout is captured")
        print("FAIL this stderr is captured", file=sys.stderr)
        logger.warning("FAIL this log is captured")
        with capsys.disabled():
            print("FAIL stdout not captured, going directly to sys.stdout")
            print("FAIL stderr not captured, going directly to sys.stderr", file=sys.stderr)
            logger.warning("FAIL is this log captured?")
        print("FAIL this stdout is also captured")
        print("FAIL this stderr is also captured", file=sys.stderr)
        logger.warning("FAIL this log is also captured")
>       assert False
E       assert False

tests/test_pytest_fold_1.py:136: AssertionError
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
FAIL this log is captured
FAIL this log is captured
FAIL is this log captured?
FAIL is this log captured?
FAIL this log is also captured
FAIL this log is also captured
FAIL this stdout is captured
FAIL this stdout is also captured
---------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------
FAIL this stderr is captured
FAIL this stderr is also captured
----------------------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------------------
WARNING  root:test_pytest_fold_1.py:128 FAIL this log is captured
WARNING  root:test_pytest_fold_1.py:132 FAIL is this log captured?
WARNING  root:test_pytest_fold_1.py:135 FAIL this log is also captured
_________________________________________________________________________________ test_11_pass_capturing _________________________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x10cce1870>

    def test_11_pass_capturing(capsys):
        print("\nPASS this stdout is captured")
        print("PASS this stderr is captured", file=sys.stderr)
        logger.warning("PASS this log is captured")
>       with capsys.disabled(log_testname):

tests/test_pytest_fold_1.py:143:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.pyenv/versions/3.10.2/lib/python3.10/contextlib.py:281: in helper
    return _GeneratorContextManager(func, args, kwds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <contextlib._GeneratorContextManager object at 0x10cce1f90>, func = <function CaptureFixture.disabled at 0x10c644790>
args = (<_pytest.capture.CaptureFixture object at 0x10cce1870>, <function log_testname at 0x10d73ac20>), kwds = {}

    def __init__(self, func, args, kwds):
>       self.gen = func(*args, **kwds)
E       TypeError: CaptureFixture.disabled() takes 1 positional argument but 2 were given

../../.pyenv/versions/3.10.2/lib/python3.10/contextlib.py:103: TypeError
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
PASS this log is captured
PASS this log is captured

PASS this stdout is captured
---------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------
PASS this stderr is captured
----------------------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------------------
WARNING  root:test_pytest_fold_1.py:142 PASS this log is captured
______________________________________________________________________________ test_12_fails_and_has_stdout ______________________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x10cd43ac0>

    def test_12_fails_and_has_stdout(capsys):
        print("this test fails")
>       assert 0 == 1
E       assert 0 == 1

tests/test_pytest_fold_1.py:155: AssertionError
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
this test fails
______________________________________________________________________________________ test_ii_fail ______________________________________________________________________________________

    def test_ii_fail():
>       assert 0
E       assert 0

tests/test_pytest_fold_2.py:25: AssertionError
_________________________________________________________________________________ test_one_warning_fails _________________________________________________________________________________

    def test_one_warning_fails():
        warnings.warn(Warning(fake_data(50,200)))
        warnings.warn(UserWarning(fake_data(55,205)))
        warnings.warn(DeprecationWarning(fake_data(55,205)))
        warnings.warn(SyntaxWarning(fake_data(55,205)))
        warnings.warn(RuntimeWarning(fake_data(55,205)))
        warnings.warn(FutureWarning(fake_data(55,205)))
        warnings.warn(PendingDeprecationWarning(fake_data(55,205)))
        warnings.warn(ImportWarning(fake_data(55,205)))
        warnings.warn(UnicodeWarning(fake_data(55,205)))
        warnings.warn(BytesWarning(fake_data(55,205)))
        warnings.warn(ResourceWarning(fake_data(55,205)))
        warnings.warn((fake_data(55,205)))
>       assert False
E       assert False

tests/test_pytest_fold_warning.py:21: AssertionError
==================================================================================== warnings summary ====================================================================================
tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:9: Warning: Car along social deep concern memory. Call surface might its stop. History firm once act tree study. Hotel suffer learn wrong show actually evidence.
    warnings.warn(Warning(fake_data(50,200)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:10: UserWarning: Plan put vote. International use thought certainly this energy look. Hand prove material issue these first take tough.
    warnings.warn(UserWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:11: DeprecationWarning: Leader area rest card after next.
    warnings.warn(DeprecationWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:12: SyntaxWarning: Particularly all since that college. Even none spring not music owner stop.
    warnings.warn(SyntaxWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:13: RuntimeWarning: Push building war modern choose very view surface.
    warnings.warn(RuntimeWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:14: FutureWarning: Common night wear fish war. Account hope western. Go station seven military.
    warnings.warn(FutureWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:15: PendingDeprecationWarning: Everyone later film possible. Picture Democrat soon time.
    warnings.warn(PendingDeprecationWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:17: UnicodeWarning: To woman likely stuff different both half. Make run fall actually carry week.
    warnings.warn(UnicodeWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:18: BytesWarning: Class either sense fall parent agreement. Wrong sit understand democratic gas reach behavior.
    warnings.warn(BytesWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:19: ResourceWarning: Than college candidate contain likely sport fire.
    warnings.warn(ResourceWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_one_warning_fails
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:20: UserWarning: Miss rather visit bag show find baby. Society commercial wall later.
    warnings.warn((fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:24: Warning: Mean full fine hope song recent guess.
    warnings.warn(Warning(fake_data(50,200)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:25: UserWarning: Street realize why time energy. Gun collection center.
    warnings.warn(UserWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:26: DeprecationWarning: No writer response man ready one goal where. Particularly door few court store commercial can.
    warnings.warn(DeprecationWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:27: SyntaxWarning: Record oil chance fight themselves really list citizen. Option six occur will energy.
    warnings.warn(SyntaxWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:28: RuntimeWarning: Message beyond significant matter cell daughter offer. Agency city against item fish. Small perhaps lose maintain.
  Ask respond assume.
    warnings.warn(RuntimeWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:29: FutureWarning: Her fight continue century none. Then old catch chance.
    warnings.warn(FutureWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:30: PendingDeprecationWarning: Around so science line off beautiful million. Option suddenly action prevent child while. Analysis employee only mean.
    warnings.warn(PendingDeprecationWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:32: UnicodeWarning: Candidate seven future recent. There herself year Mr thus statement leave.
  Involve list trade. Man season attention newspaper. Become agree home current quite day.
    warnings.warn(UnicodeWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:33: BytesWarning: After sense his result. Argue forget then sure. Forward herself picture including marriage paper positive. I its effect identify may oil here rock.
    warnings.warn(BytesWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:34: ResourceWarning: Public decision arm our.
  Shake risk worker know board half.
    warnings.warn(ResourceWarning(fake_data(55,205)))

tests/test_pytest_fold_warning.py::test_two_warning_passes
  /Users/jwr003/coding/pytest-fold/tests/test_pytest_fold_warning.py:35: UserWarning: Hold recently you ok recognize. Man know mean fly how effect resource long.
    warnings.warn((fake_data(55,205)))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================= PASSES =========================================================================================
_______________________________________________________________________________________ test_a_ok ________________________________________________________________________________________
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
ok
__________________________________________________________________________ test_1_passes_and_has_logging_output __________________________________________________________________________
--------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------
Running test tests.test_pytest_fold_1...
Running test tests.test_pytest_fold_1...
Setting test up...
Setting test up...
Executing test...
Executing test...
Tearing test down...
Tearing test down...
----------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------
INFO     root:test_pytest_fold_1.py:52 Running test tests.test_pytest_fold_1...
INFO     root:test_pytest_fold_1.py:53 Setting test up...
INFO     root:test_pytest_fold_1.py:54 Executing test...
INFO     root:test_pytest_fold_1.py:56 Tearing test down...
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
Whole forward blue type adult that follow feeling. Example interview beat kind.
Whole forward blue type adult that follow feeling. Example interview beat kind.
Peace but modern respond who its.
Peace but modern respond who its.
Attack certain here think American. Drive but drug authority before.
Attack certain here think American. Drive but drug authority before.
Sound we future.
Sound we future.
Health table indicate environment have area book admit.
Health table indicate environment have area book admit.
----------------------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------------------
CRITICAL root:test_pytest_fold_1.py:63 Whole forward blue type adult that follow feeling. Example interview beat kind.
ERROR    root:test_pytest_fold_1.py:64 Peace but modern respond who its.
WARNING  root:test_pytest_fold_1.py:65 Attack certain here think American. Drive but drug authority before.
INFO     root:test_pytest_fold_1.py:66 Sound we future.
DEBUG    root:test_pytest_fold_1.py:67 Health table indicate environment have area book admit.
_____________________________________________________________________________________ test_4_passes ______________________________________________________________________________________
--------------------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------------------
Running test tests.test_pytest_fold_1...
Running test tests.test_pytest_fold_1...
Setting test up...
Setting test up...
Executing test...
Executing test...
Tearing test down...
Tearing test down...
----------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------
INFO     root:test_pytest_fold_1.py:52 Running test tests.test_pytest_fold_1...
INFO     root:test_pytest_fold_1.py:53 Setting test up...
INFO     root:test_pytest_fold_1.py:54 Executing test...
INFO     root:test_pytest_fold_1.py:56 Tearing test down...
_____________________________________________________________________________ test_13_passes_and_has_stdout ______________________________________________________________________________
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
this test passes
_______________________________________________________________________________________ test_i_ok ________________________________________________________________________________________
---------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
ok
================================================================================ short test summary info =================================================================================
PASSED tests/test_pytest_fold_1.py::test_a_ok
PASSED tests/test_pytest_fold_1.py::test_g_eval_parameterized[3+5-8]
PASSED tests/test_pytest_fold_1.py::test_g_eval_parameterized[2+4-6]
PASSED tests/test_pytest_fold_1.py::test_1_passes_and_has_logging_output
PASSED tests/test_pytest_fold_1.py::test_4_passes
PASSED tests/test_pytest_fold_1.py::test_13_passes_and_has_stdout
PASSED tests/test_pytest_fold_2.py::test_i_ok
PASSED tests/test_pytest_fold_warning.py::test_two_warning_passes
SKIPPED [1] tests/test_pytest_fold_1.py:34: skipping this test
SKIPPED [1] tests/test_pytest_fold_1.py:88: unconditional skip
SKIPPED [1] tests/test_pytest_fold_2.py:33: skipping this test
XFAIL tests/test_pytest_fold_1.py::test_e_xfail
  reason: xfailing this test
XFAIL tests/test_pytest_fold_1.py::test_7_marked_xfail_and_fails
XFAIL tests/test_pytest_fold_2.py::test_v_xfail
  reason: xfailing this test
XPASS tests/test_pytest_fold_1.py::test_f_xpass always xfail
XPASS tests/test_pytest_fold_1.py::test_6_marked_xfail_but_passes
XPASS tests/test_pytest_fold_2.py::test_vi_xpass always xfail
ERROR tests/test_pytest_fold_1.py::test_c_error - assert 0
ERROR tests/test_pytest_fold_1.py::test_14_causes_error_pass_stderr_stdout_stdlog
ERROR tests/test_pytest_fold_1.py::test_15_causes_error_fail_stderr_stdout_stdlog
ERROR tests/test_pytest_fold_2.py::test_iii_error - assert 0
FAILED tests/test_pytest_fold_1.py::test_b_fail - assert 0
FAILED tests/test_pytest_fold_1.py::test_g_eval_parameterized[6*9-42] - AssertionError: assert 54 == 42
FAILED tests/test_pytest_fold_1.py::test_2_fails_and_has_logging_output - assert 0 == 1
FAILED tests/test_pytest_fold_1.py::test_3_fails - assert 0
FAILED tests/test_pytest_fold_1.py::test_8_causes_a_warning - TypeError: api_v1() missing 1 required positional argument: 'log_testname'
FAILED tests/test_pytest_fold_1.py::test_9_lorem_fails - assert False
FAILED tests/test_pytest_fold_1.py::test_10_fail_capturing - assert False
FAILED tests/test_pytest_fold_1.py::test_11_pass_capturing - TypeError: CaptureFixture.disabled() takes 1 positional argument but 2 were given
FAILED tests/test_pytest_fold_1.py::test_12_fails_and_has_stdout - assert 0 == 1
FAILED tests/test_pytest_fold_2.py::test_ii_fail - assert 0
FAILED tests/test_pytest_fold_warning.py::test_one_warning_fails - assert False
================================================== 11 failed, 8 passed, 3 skipped, 3 xfailed, 3 xpassed, 22 warnings, 4 errors in 3.20s ==================================================
(pytest-fold)
~/coding/pytest-fold on  main! ⌚ 7:27:10
$

1 reply

jeffwright13 Apr 4, 2022
Author

Testfiles that create the above output are available at https://github.com/jeffwright13/pytest-fold/tree/main/tests.

okken · 2023-07-19T16:03:17Z

okken
Jul 19, 2023
Collaborator

I'm running into this problem also.

Example: test_foo.py

import pytest

def test_pass():
    print('in test_pass()')
    a, b = 1, 1
    assert a == b

def test_fail():
    print('in test_fail()')
    a, b = 1, 2
    assert a == b

@pytest.mark.xfail
def test_xfail():
    print('in test_xfail()')
    a, b = 1, 2
    assert a == b

@pytest.mark.xfail
def test_xpass():
    print('in test_xpass()')
    a, b = 1, 1
    assert a == b

Current output:

$ pytest -rA test_foo.py
============================= test session starts =============================
platform win32 -- Python 3.11.0, pytest-7.4.0, pluggy-1.0.0
rootdir: C:\Users\okken\projects\instrument_updater
configfile: tox.ini
collected 4 items

test_foo.py .FxX                                                         [100%]

================================== FAILURES ===================================
__________________________________ test_fail __________________________________

    def test_fail():
        print('in test_fail()')
        a, b = 1, 2
>       assert a == b
E       assert 1 == 2

test_foo.py:11: AssertionError
---------------------------- Captured stdout call -----------------------------
in test_fail()
=================================== PASSES ====================================
__________________________________ test_pass __________________________________
---------------------------- Captured stdout call -----------------------------
in test_pass()
=========================== short test summary info ===========================
PASSED test_foo.py::test_pass
XFAIL test_foo.py::test_xfail
XPASS test_foo.py::test_xpass
FAILED test_foo.py::test_fail - assert 1 == 2
============== 1 failed, 1 passed, 1 xfailed, 1 xpassed in 0.28s ==============

Observations:

Output and exception for test_fail but not for test_xfail.
- I would have expected test_xfail to look mostly just like test_fail.
- If there's a reason to NOT report exception traceback and output for xfails, we should have an option to turn it on.
assert 1 == 2 is displayed in summary info for FAILED but not for XFAIL.
- I can't come up with reason why this would be correct behavior.
- assert 1 == 2 should show up for XFAIL also.
-rP (which is included in -rA) is "pass with output", and it applies to PASSED, but not XPASS.

Opinion on how to fix this:

XFAIL should act like FAILED if it's turned on with -rx or -ra or -rA
- output should be reported
- traceback should be reported
- assert message should be listed in summary
XPASS should act like PASSED if it's turned on with -rX or -ra or -rA
- output should be reported
- traceback should be reported
- assert message should be listed in summary
It seems reasonable that someone might not want to see all of this extra output.
- That's why I've suggested that this extra output NOT be on by default.
- Controlling the extra output with -r flags seems like the right way to do this.

My opinions, of course. But this would follow the idea of "behavior with the least surprise". And it doesn't require any extra flags.

1 reply

okken Feb 8, 2024
Collaborator

Most of this has been implemented in pytest 8.0.0

Michael1248 · 2024-02-08T16:47:10Z

Michael1248
Feb 8, 2024

(I had the same problem) its hacky but its a solution

@pytest.mark.xfail(raises=AssertionError, reason="a reason")
def test_a():
    try:
        assert 1 == 2
    except AssertionError as e:
        print(e.args[0])
        raise e

and execute pytest with -s option:

$ pytest -s -vv
[...]
assert 1 == 2
  +1
  -2
XFAIL (a reason)
[...]

-s shows you the printed stuff in cmd, and -vv does show longer output.

EDIT:

$ pytest -V
pytest 6.2.4

1 reply

okken Feb 8, 2024
Collaborator

pytest 8.0.0 reports output for xpass and xfail if you use -rxX or -ra (which includes -rx for xfail and -rX for xpass )

okken · 2024-02-08T17:03:16Z

okken
Feb 8, 2024
Collaborator

pytest 8.0.0 reports ouptut for xpass and xfail if you use -rxX or -ra (which includes -rx for xfail and -rX for xpass )

1 reply

jorisvandenbossche Feb 13, 2024

With pytest 8.0, there are now indeed separate sections with the verbose output when enabled (-r xX or -r a). But is there still a way to see the xfails/xpasses in the "short test summary info" listing?

It seems the -r flag controls both (separate section + being listed in summary), while in my experience, the full output is too verbose in many cases to enable this flag by default in CI (it would make it harder to look at actual failures), but I would still like to see the xfailed/xpassed tests listed in the short summary in the end.
That kind of configuration doesn't seem to be possible at the moment?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why no output sections for Xpass/Xfail? #9834

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Why no output sections for Xpass/Xfail? #9834

jeffwright13 Apr 3, 2022

Replies: 5 comments · 7 replies

nicoddemus Apr 4, 2022 Maintainer

jeffwright13 Apr 4, 2022 Author

nicoddemus Apr 4, 2022 Maintainer

jeffwright13 Apr 4, 2022 Author

jeffwright13 Apr 4, 2022 Author

jeffwright13 Apr 4, 2022 Author

okken Jul 19, 2023 Collaborator

okken Feb 8, 2024 Collaborator

Michael1248 Feb 8, 2024

okken Feb 8, 2024 Collaborator

okken Feb 8, 2024 Collaborator

jorisvandenbossche Feb 13, 2024

jeffwright13
Apr 3, 2022

Replies: 5 comments 7 replies

nicoddemus
Apr 4, 2022
Maintainer

jeffwright13 Apr 4, 2022
Author

nicoddemus Apr 4, 2022
Maintainer

jeffwright13 Apr 4, 2022
Author

jeffwright13
Apr 4, 2022
Author

jeffwright13 Apr 4, 2022
Author

okken
Jul 19, 2023
Collaborator

okken Feb 8, 2024
Collaborator

Michael1248
Feb 8, 2024

okken Feb 8, 2024
Collaborator

okken
Feb 8, 2024
Collaborator