doc: improve testing guide #11150

joyeecheung · 2017-02-03T18:18:51Z

Add guide on choice of assertions, use of ES.Next features, and the WPT upstream.

Refs: #11142

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

doc

Add guide about choice of assertions and encourage using ES.Next features.

Trott

LGTM

I left a bunch of style nits, but they are all optional. I'm fine with this as is, and I'm fine with it with any or all of the changes I suggested.

Trott · 2017-02-03T21:37:38Z

doc/guides/writing-tests.md

+### Assertions
+
+When writing assertions for object comparison, prefer the strict versions,
+namely:


Nit: Get rid of namely.

...prefer the strict versions:

assert.strictEqual()...

Trott · 2017-02-03T21:38:20Z

doc/guides/writing-tests.md

@@ -209,6 +209,36 @@ const assert = require('assert');
 const freelist = require('internal/freelist');
 ```

+### Assertions
+
+When writing assertions for object comparison, prefer the strict versions,


Nit: Remove object.

When writing assertions for comparisons, prefer the strict versions:

Trott · 2017-02-03T21:38:58Z

doc/guides/writing-tests.md

+### ES.Next features
+
+At the moment we only use a selected subset of ES.Next features in
+JavaScript code under the `lib` directory for performance considerations,


Nit: under -> in

Trott · 2017-02-03T21:39:34Z

doc/guides/writing-tests.md

+### ES.Next features
+
+At the moment we only use a selected subset of ES.Next features in
+JavaScript code under the `lib` directory for performance considerations,


Nit: Remove At the moment and move for performance considerations to the beginning:

For performance considerations, we only use a selected subset...

Trott · 2017-02-03T21:40:19Z

doc/guides/writing-tests.md

+
+* `let` and `const` over `var`
+* Template literals over string concatenation
+* Arrow functions over normal functions, if appropriate


Nit:

Arrow functions when appropriate

Trott · 2017-02-03T21:40:52Z

doc/guides/writing-tests.md

+
+At the moment we only use a selected subset of ES.Next features in
+JavaScript code under the `lib` directory for performance considerations,
+but when writing tests, it is encouraged to use ES.Next features that have


Nit: Make this the start of its own sentence and change but to However.

However, when writing tests, it is...

Trott · 2017-02-03T21:41:42Z

doc/guides/writing-tests.md

+Some of the tests for the WHATWG URL implementation (named
+`test-whatwg-url-*.js`) are imported from the
+[Web Platform Tests Project](https://github.com/w3c/web-platform-tests/tree/master/url).
+If you see a test wrapped in code like this


Nit: Replace this line with:

These imported tests will be wrapped like this:

Trott · 2017-02-03T21:42:26Z

doc/guides/writing-tests.md

+/* eslint-enable */
+```
+
+and want to improve it, please send a PR to the upstream project


Nit: Have this be the start of its own sentence.

If you want to improve tests that have been imported this way, please send a...

Trott · 2017-02-03T21:43:15Z

doc/guides/writing-tests.md

+```
+
+and want to improve it, please send a PR to the upstream project
+first, and when it's merged, send another PR to this project


Nit: End the sentence after first and begin a new sentence after that:

first. When your proposed change is merged in the upstream project, send another PR to...

Trott · 2017-02-03T21:43:51Z

doc/guides/writing-tests.md

+
+and want to improve it, please send a PR to the upstream project
+first, and when it's merged, send another PR to this project
+to update the code accordingly (be sure to update the hash in the


Nit:

to update the code accordingly. Be sure to update the hash in the...

(Also, if you do that, remove the closing parenthesis on the next line.)

Trott · 2017-02-03T21:45:29Z

@nodejs/testing @nodejs/documentation

joyeecheung · 2017-02-04T13:53:53Z

@Trott Thanks for the review, PTAL.

Trott · 2017-02-04T20:14:12Z

doc/guides/writing-tests.md

@@ -209,6 +209,35 @@ const assert = require('assert');
 const freelist = require('internal/freelist');
 ```

+### Assertions
+
+When writing assertions for comparison, prefer the strict versions:


Nit: Probably clearer to have singular/plural agreement: writing an assertion for comparison or writing assertions for comparisons.

Trott · 2017-02-04T20:15:59Z

doc/guides/writing-tests.md

+
+If you want to improve tests that have been imported this way, please send
+a PR to the upstream project first. When your proposed change is merged in
+the upstream project, send another PR to update the code accordingly.


Nit: another PR to update Node.js accordingly or something else that indicates that the second PR should be against nodejs/node and not against the upstream project.

Trott · 2017-02-04T20:16:22Z

doc/guides/writing-tests.md

+If you want to improve tests that have been imported this way, please send
+a PR to the upstream project first. When your proposed change is merged in
+the upstream project, send another PR to update the code accordingly.
+Be sure to update the hash in the hash in the URL following `WPT Refs:`.


Looks like there's one too many hash in the.

joyeecheung · 2017-02-06T03:34:42Z

@Trott Updated, PTAL.

thefourtheye · 2017-02-06T04:28:17Z

doc/guides/writing-tests.md

@@ -209,6 +209,35 @@ const assert = require('assert');
 const freelist = require('internal/freelist');
 ```

+### Assertions
+
+When writing an assertions for comparison, prefer the strict versions:


an assertions should be an assertion or simply assertions.

Oops, thanks for catching that.

thefourtheye · 2017-02-06T05:22:05Z

doc/guides/writing-tests.md

@@ -209,6 +209,35 @@ const assert = require('assert');
 const freelist = require('internal/freelist');
 ```

+### Assertions
+
+When writing an assertion for comparison, prefer the strict versions:


Strictly speaking, assertion is not for comparison.

Hmmm, yes, the word "for" does sound wrong here, or the order of assertion and comparison here is wrong. Probably assertion with comparison?

You can simply drop for comparison.. "When asserting, prefer strict versions" - How does it sound?

Yeah I think that works too.

thefourtheye · 2017-02-06T06:30:16Z

doc/guides/writing-tests.md

+* `assert.strictEqual()` over `assert.equal()`
+* `assert.deepStrictEqual()` over `assert.deepEqual()`
+
+When using `assert.throws()`, if possible, provide the full error message:


We could encourage people to always use the full message, so that we can avoid incidents like #11162

So this would better be "provide the full error message unless it is impossible"? (I am trying to think of a scenario where it is impossible, which I believe I've seen before, but nothing come out of my head at the moment)

@Trott Any thoughts, how to handle this?

I'm OK with the text the way it is. We can always improve it in a subsequent pull request if someone comes up with better wording. I think "if possible, provide the full error message" is about as good as anything I can come up with right now....

If you want to drop "if possible" and just say "provide the full error message", that works for me too. I have no strong opinion either way. Adding either one would be an improvement to the doc, so 👍 from me whichever way you go on it.

well... I wouldn't say impossible, but still requires a lot of work, right now I am having a hard time with this problem here: #11096 , error messages change across platforms, so I would say make the suggestion but put a note saying that this could happen

Or should I just say: the regular expression used to validate error messages should be as concrete as possible?

Or

When using `assert.throws()`, prefer using regular expressions to validate error messages, and the regular expressions provided should be as concrete as possible:

It may just be terminology I'm unfamiliar with, but without the context of this conversation, I wouldn't know what this would mean. If someone told me, "That's pretty good, but could you make the regular expression more concrete?" ... I would have to ask what they meant.

I think the existing wording in this PR is the best we've managed to come up with yet. I'm OK with leaving it as is and saving any bike-shedding for subsequent revisions, but if we want to tweak it now maybe this?:

When using assert.throws(), the second argument should be either a regular expression or a function that validates the received error. If using a regular expression, make the match as specific as possible. Match the entire error string by starting the regular expression with ^ and ending with $.

It's a little bit more wordy, but this should be less confusing to new contributors so I think it's worth it.

I'll go ahead and land this PR first, and make the improvement a good first contributon issue linking to this discussion, so we can get a fresh angle from a new contributor.

joyeecheung · 2017-02-07T15:39:46Z

CI: https://ci.nodejs.org/job/node-test-pull-request/6257/

joyeecheung · 2017-02-07T15:49:17Z

Linter is green, landed in 7ba847d

Add guide on choice of assertions, use of ES.Next features, and the WPT upstream. PR-URL: nodejs#11150 Ref: nodejs#11142 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Santiago Gimeno <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Timothy Gu <[email protected]>

Add guide on choice of assertions, use of ES.Next features, and the WPT upstream. PR-URL: #11150 Ref: #11142 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Santiago Gimeno <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Timothy Gu <[email protected]>

joyeecheung added 2 commits February 4, 2017 02:14

doc: update test writing guide

47235f0

Add guide about choice of assertions and encourage using ES.Next features.

doc: update testing guide regarding WPT upstream

4f6f235

nodejs-github-bot added the doc Issues and PRs related to the documentations. label Feb 3, 2017

joyeecheung requested a review from Trott February 3, 2017 18:21

mscdex added the test Issues and PRs related to the tests. label Feb 3, 2017

Trott approved these changes Feb 3, 2017

View reviewed changes

fix style

1d9c9f0

santigimeno approved these changes Feb 4, 2017

View reviewed changes

jasnell approved these changes Feb 4, 2017

View reviewed changes

TimothyGu approved these changes Feb 4, 2017

View reviewed changes

Trott reviewed Feb 4, 2017

View reviewed changes

fix styles

f236299

thefourtheye reviewed Feb 6, 2017

View reviewed changes

fix typo

7ce8f5d

thefourtheye reviewed Feb 6, 2017

View reviewed changes

drop 'for comparison'

61fb2e8

thefourtheye reviewed Feb 6, 2017

View reviewed changes

joyeecheung mentioned this pull request Feb 6, 2017

test: add regex #11193

Closed

3 tasks

joyeecheung closed this Feb 7, 2017

italoacasas mentioned this pull request Feb 13, 2017

Proposal - 7.6.0 (Current) #11185

Merged

joyeecheung deleted the testing-guide branch February 19, 2017 17:42

MylesBorins mentioned this pull request Mar 9, 2017

v6.10.1 proposal #11759

Merged

MylesBorins mentioned this pull request Mar 9, 2017

v4.8.1 proposal #11760

Merged

doc: improve testing guide #11150

doc: improve testing guide #11150

Conversation

joyeecheung commented Feb 3, 2017 • edited Loading

Checklist

Affected core subsystem(s)

Trott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Trott commented Feb 3, 2017

joyeecheung commented Feb 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung commented Feb 6, 2017

thefourtheye Feb 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Feb 7, 2017 • edited Loading

Choose a reason for hiding this comment

Trott Feb 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung commented Feb 7, 2017

joyeecheung commented Feb 7, 2017

joyeecheung commented Feb 3, 2017 •

edited

Loading

thefourtheye Feb 6, 2017 •

edited

Loading

joyeecheung Feb 7, 2017 •

edited

Loading

Trott Feb 7, 2017 •

edited

Loading