XUnit reporter should handle exceptions during diff generation

[rgroothuijsen]

Description of the Change

This bugfix addresses a crash in the XUnit reporter, which produced no output because the assertion library used in the ticket provided numerical diff values rather than string values, causing an error during diff generation. To safeguard against unexpected crashes such as these, the same input-checks that the Base reporter uses have been implemented in the XUnit reporter, and exceptions during XUnit diff generation are returned by the reporter.

Applicable issues

Fixes #4022.

[jsf-clabot]

All committers have signed the CLA.

[coveralls]

Coverage increased (+0.01%) to 92.913% when pulling a43def2 on rgroothuijsen:MOCHA-4022 into d9f5079 on mochajs:master.

[craigtaub]

Managed to replicate myself. The fix LGTM. @mochajs/core any other thoughts?

[juergba]

When this exception is thrown in XUnit reporter, the output is truncated and aborted without any error message. That exception shouldn't be swallowed.

Could be related to #980.

[rgroothuijsen]

@juergba I'm not sure if I understood entirely. Is this particular fix incorrect? Is there something I should add/modify?

[juergba]

This issue has two parts. The diff exception and then second the reporter aborting without any message.
It would be helpful to know the cause of the second part in order to decide wether this PR is complete.

[rgroothuijsen]

@juergba Uncaught exceptions are intercepted by the runner at this point. Looking at the method in question, I see that the exception is logged, but only at the debug log level.

[juergba]

@rgroothuijsen thank you.

I'm not happy mainly with the existing implementation of diff in XUnit reporter.

• Base.generateDiff: exceptions are not catched and propagate upto the uncaughtException handler.
• condition when to generate diff is much softer in XUnit than in Base. eg. there is no check for identical type of actual and expected.
• I'm unsure wether this PR is IE11 compatible.

[juergba]

We have an EVENT_TEST_FAIL handler in Base which calls stringifyDiffObjs. Since XUnit extends Base, this handler is called also for XUnit.
So the problem is not the missing stringifyDiffObjs in XUnit, but the conditions when to generate a diff, are not identical in Base and XUnit.

[juergba]

I added some change requests to your code:

• we catch an exception while generating diff
• we stringify only in the fail event handler
• we synchronize conditions when diffs should be generated over Base, XUnit and fail event handler.

I haven't tested my proposition, so if you find anything, please go ahead ...

[juergba]

Those mocking tests are quite complex, so please check my code carefully.

[juergba]

@rgroothuijsen thank you!
Could you please rebase, squash with your commit message and edit/update this PR's description?

[rgroothuijsen]

@juergba Will do. In retrospect, do you think the title still works for this PR?

[juergba]

@rgroothuijsen No, not really. Maybe "XUnit reporter: fix crash during diff generation" or similar, but the title is upto you.