jest-circus ordering and error-handling behavior undocumented

[cspotcode]

🐛 Bug Report

I can't find docs explaining execution order of suites, cases, and hooks in jest-circus. Docs also don't explain error handling behavior for failed hooks. This information is necessary for integration and end-to-end test suites which have expensive, slow setup and teardown operations.

I'm evaluating jest for integration and E2E suites that currently run on mocha. I'd like to move to jest for extra features like --json. However, execution ordering is undocumented, so I'm not sure if tests and hooks will run in the correct order, nor how hook failures will be handled.

I'm willing to submit bugfixes to jest-circus, but I can't do that if I don't know the intended, desired behavior of jest-circus. What are the team's goals?

• Do test cases run in declaration order?
• What is the sorting of peer suites and cases?
• Do failed hooks report their failures as hook failures?
• When a hook fails, are dependent suites and cases properly skipped and marked failed because of the hook?
• When --runInBand is enabled, will suites and cases run 100% serially? When it's disabled, will all suites and cases within a single file run 100% serially?

This doc does not need to be a part of the website; it could be a markdown file in ./packages/jest-circus tracking the current desired behavior. This way, I can use the document to prove my team can or can't use jest, and I can submit bugfixes to make jest-circus align with the document.

Related: #8360, #2713

[cspotcode]

Any update on this?

[jeysal]

@cspotcode #8360 was fixed in Jest 26 - I think there is now no execution order difference between circus and Jasmine, and I think I'm finally confident to say that circus is approaching readiness to become the default (Jest 27). I'd say now that it is the same as Jasmine, the need for documentation around it is a bit smaller, and I'd like to avoid it in order to not encourage writing order-dependent tests. (Although I'm aware people still do it e.g. for e2e tests and I'm not saying it's a generally invalid approach)

[cspotcode]

What is the plan to internally document that ordering must be preserved, so that future changes do not break it? The justification for omitting docs about ordering is awfully flimsy.

…

On Mon, May 4, 2020, 4:16 PM Tim Seckinger ***@***.***> wrote: @cspotcode <https://github.com/cspotcode> #8360 <#8360> was fixed in Jest 26 - I think there is now no execution order difference between circus and Jasmine, and I think I'm finally confident to say that circus is approaching readiness to become the default (Jest 27). I'd say now that it is the same as Jasmine, the need for documentation around it is a bit smaller, and I'd like to avoid it in order to not encourage writing order-dependent tests. (Although I'm aware people still do it e.g. for e2e tests and I'm not saying it's a generally invalid approach) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9446 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OBWCJPC3LDCULA3N33RP4PBNANCNFSM4KKL7VWA> .

[jeysal]

Every significant change, including the one that fixed #8360, must contain sufficient regression tests.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.