esm: Implement esm mode flag #18392

guybedford · 2018-01-26T11:07:34Z

This provides a mode: "esm" flag for package.json files which will treat ".js" files as ES modules within a given package boundary.

The package boundary of a package.json file is deemed to be all modules in that folder and subfolder until the next nested package.json file.

The plan is to add a top-level --mode flag to this, roughly matching the npm proposal, but restricted to the current package boundary. This work is currently at https://github.com/guybedford/node/tree/package-top-mode.

The mode effectively marks a directory as an "esm mode" directory, where all ".js" files in that directory are loaded as ES modules, unless they are located in a subfolder that contains a package.json without an "esm mode", which results in reverting to the transparent interop of treating ".js" as CommonJS.

In addition this PR enables caching of package.json lookups within the ES module resolver.

Presentation: https://docs.google.com/presentation/d/1xK1ana_TIxfAHX33CYVHFnJsV0If_YirLtRBBga9cv0/edit#slide=id.p

This work is based on the following prior proposal work:

package.json "esm" flag (adjusted proposal: ES module "esm": true package.json flag node-eps#60), where this replaces the PR from esm: Implement package.json "esm": true flag for ES modules in ".js" files #18156
npm's --mode proposal (https://gist.github.com/ceejbot/b49f8789b2ab6b09548ccb72813a1054#parse-goals)
package mime goals (https://gist.github.com/bmeck/7ee7eb2147e2dafe3167c856d9b4151a)

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

esmodules

bmeck · 2018-01-26T14:19:32Z

I have a writeup from conversations in form of a pseudo design document.

jdalton · 2018-01-26T16:43:44Z

lib/internal/loader/DefaultResolve.js

@@ -39,14 +43,22 @@ function search(target, base) {
  }
 }

-const extensionFormatMap = {
+const extensionFormatStd = {
  __proto__: null,
  '.mjs': 'esm',
  '.json': 'json',
  '.node': 'addon',
  '.js': 'commonjs'
 };


☝️ still not a fan of using short forms for most names but not commonjs. We have ECMAScript module -> esm, JavaScript Object Notation -> json, CommonJS -> should be cjs.

tbh I have no idea why I made this change and if it had been brought up before my PR landed I would have most likely changed it back, kinda feelin' guilty about this one.

Babel now allows specifying "cjs", in addition to "commonjs", as their module mode because folks kept reaching for it.

I would also like to see it changed back to "cjs"

jdalton · 2018-01-26T16:48:45Z

src/node.cc

+      }
+      config_main_mode = arg + 7;
+      if (config_main_mode != "esm") {
+        fprintf(stderr, "%s: \"%s\" is not a valid module mode\n", argv[0],


👆 I'm not a pro in c++ but is config_main_mode = arg + 7 a number result? If so would it never match "esm" in the if statement.

config_main_mode is a std::string, so comparing using the operator works. :) I assume arg + 7 is the pointer to the null-terminated string starting 7 bytes from the pointer arg, skipping the --mode= (7 bytes). But it's not the most straight-forward to follow.

jdalton · 2018-01-26T16:50:52Z

src/node.cc

+      args_consumed += 1;
+      config_main_mode = main_mode;
+      if (config_main_mode != "esm") {
+        fprintf(stderr, "%s: \"%s\" is not a valid module mode\n", argv[0],


👆 If there is two modes documented shouldn't both modes be allowed?

it does seem like you should be able to set "commonjs" or "cjs" as the mode

jdalton · 2018-01-26T16:52:23Z

The mode effectively marks a directory as an "esm mode" directory, where all ".js" files

What about extensionless bin files that CLI's often use?

Update

There's a mention in the writeup about --format related to extensionless but I'm not sure why a mode property couldn't cover it.

The writeup mentions a gotcha with require and extensionless files.
Does this mean require can load ESM?

JacopKane · 2018-01-26T17:08:18Z

Thanks a lot for your work that might save us from a new javascript extension.

I know this is an entirely minor detail to consider right now, but I would only argue using shorthand, esm, instead of something more descriptive like esModule, ecmaModule or maybe something else but just more apparent.

bmeck · 2018-01-26T17:16:12Z

@jdalton --format is for entry points that don't have extensions at all / STDIN, it does not alter the default format of file extensions. If --mode affected those situations it would need to expand scope so that all extension-less files were treated in a specific way. This complication is why it was punted.

jdalton · 2018-01-26T17:19:14Z

@bmeck

This complication is why it was punted.

What about tackling it in the shebang of the bin file like

#!/usr/bin/env node --mode esm

bmeck · 2018-01-26T17:22:05Z

@jdalton punted and as describe in the design doc "bin" files can have extensions even if the command does not which is the common case for using command line wrappers, feel free to PR the --format flag as described if you feel strongly about needing that specific use case.

jdalton · 2018-01-26T17:25:08Z

punted and as describe in the design doc "bin" files can have extensions even if the

The bin file is opting in to esm with the --mode in the shebang. It's not executing the command. Node would parse it out. How you handle conflicting signals is a separate issue.

bmeck · 2018-01-26T17:27:41Z

@jdalton --mode is setting up file extension mapping similar to a MIME DB, --format is a very different idea which overrides something that doesn't have a well known intention already. We absolutely should not conflate these flags in my opinion.

jdalton · 2018-01-26T17:29:46Z

We absolutely should not conflate these flags in my opinion.

Having it specified in the shebang is a nice descriptive and natural way to tackle extensionless bins.

bmeck · 2018-01-26T17:30:08Z

@jdalton PR up the --format flag :)

jdalton · 2018-01-26T17:30:33Z

Naw. It doesn't have to be --format or bust.

jkrems · 2018-01-26T17:30:59Z

@jdalton One problem with putting arguments into the shebang is that it's not super portable. :( https://unix.stackexchange.com/questions/63979/shebang-line-with-usr-bin-env-command-argument-fails-on-linux

P.S.: Shipping something like a nodem or mnode shim with node might fix that though.

jdalton · 2018-01-26T17:33:05Z

Or we could try it in this PR, write tests, and see if issues are had.

jkrems · 2018-01-26T17:37:27Z

Or we could try it in a PR, write tests, and see if issues are had.

Not sure I follow - this isn't hard to verify. The following:

$ cat test.js
#!/usr/bin/env node --harmony
console.log(process.argv, process.execArgv);

Works on OSX but won't run on (all) Linux (the same without --harmony works on both):

./test.js
/usr/bin/env: node --harmony: No such file or directory

jdalton · 2018-01-26T17:58:39Z

Not sure I follow - this isn't hard to verify. The following

Many times ideas are shot down with a quick SO post without actually seeing if, in the given scenario, it is doable or an issue is able to be worked around.

Pinging @ceejbot and @chrisdickinson who had this in their npm proposal, though it was unimplemented (wondering if they ran into similar issues or if they've thought of anything to workaround the gotcha).

P.S.: Shipping something like a nodem or mnode shim with node might fix that though.

That may be it 😋. I never considered something like that. Kinda neat!

giltayar · 2018-01-26T20:11:46Z

Question: with such a proposal, how would I write a "dual mode package", one which works whether we require the package or whether we import it. This is possible in the mjs proposal, but I don't see how you can do it in this proposal.

If we go with this proposal, all library developers that want to migrate to the esm model will need to fork their modules. So we will have moment and moment-esm. lodash and lodash-esm, uuid and uuid-esm, etc...

Given that we're going to live in dual-mode world for a long time, I believe this is not viable.

bmeck · 2018-01-26T20:16:59Z

@giltayar You would still use the .mjs approach for such a behavior. This PR is ontop of .mjs and seeks to alleviate common cases that do not need such a behavior since it is already available through .mjs. A variety of situations do not need to support dual mode packages, such as top level applications that are never meant to be treated as dependencies.

devsnek · 2018-04-28T14:24:43Z

@BridgeAR i re-opened nodejs/modules#42 which should be good enough for now i suppose

This implements a package.json "mode": "esm" module boundary. When loading a module from a package with this esm mode, ".js" files are loaded as es modules. In addition package.json caching is implemented in the module lookup process, and package.json boundaries are only checked when necessary.

ChALkeR · 2019-03-01T15:08:51Z

@devsnek nodejs/modules#42 is closed, any other reference?

devsnek · 2019-03-01T15:56:48Z

@MylesBorins maybe has something, I'm not 100% sure

richardlau · 2019-03-19T02:00:22Z

Is this superseded by #26745?

GeoffreyBooth · 2019-03-19T02:01:00Z

Is this superseded by #26745?

Yes.

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Jan 26, 2018

guybedford mentioned this pull request Jan 26, 2018

esm: Implement package.json "esm": true flag for ES modules in ".js" files #18156

Closed

4 tasks

guybedford requested a review from bmeck January 26, 2018 11:09

guybedford force-pushed the package-mode branch 6 times, most recently from 862219d to cfb5d70 Compare January 26, 2018 13:53

jdalton mentioned this pull request Jan 26, 2018

implement esm option in package.json standard-things/esm#234

Closed

jdalton reviewed Jan 26, 2018

View reviewed changes

MylesBorins added the blocked PRs that are blocked by other issues or PRs. label Apr 28, 2018

MylesBorins mentioned this pull request May 14, 2018

Pluggable Loaders to support multiple use cases nodejs/modules#82

Closed

GeoffreyBooth mentioned this pull request May 20, 2018

Feature: Dual-goal packages nodejs/modules#93

Closed

GeoffreyBooth mentioned this pull request Jun 2, 2018

Encouraging users to use experimental features to reach better outcomes nodejs/modules#122

Closed

guybedford mentioned this pull request Jun 5, 2018

Reconsidering this proposal zenparsing/node-default-module-proposal#6

Closed

guybedford force-pushed the package-mode branch 2 times, most recently from d6d6913 to f61aa91 Compare June 14, 2018 19:12

bmeck mentioned this pull request Jun 21, 2018

Proposal for transparent interop implementation nodejs/modules#137

Closed

bmeck mentioned this pull request Jun 30, 2018

disambiguate "web compatibility" nodejs/modules#142

Closed

bmeck mentioned this pull request Jul 9, 2018

MIME types nodejs/modules#146

Closed

GeoffreyBooth mentioned this pull request Jul 12, 2018

Feature: ESM in .js files nodejs/modules#151

Closed

devsnek mentioned this pull request Jul 22, 2018

ESM in .js files proposals nodejs/modules#150

Closed

This was referenced Aug 17, 2018

Node.js Foundation Modules Team Meeting 2018-08-15 nodejs/modules#167

Closed

Proposals for configuring parse goals of files in --experimental-modules nodejs/modules#160

Closed

guybedford force-pushed the package-mode branch from 303d5a8 to 459ec5c Compare August 28, 2018 16:32

guybedford added 5 commits October 2, 2018 22:09

PR feedback

89bfdfa

further PR corrections

19d9295

linting fixes

a92792d

fixup rebase

9b1de85

guybedford force-pushed the package-mode branch from 0df17e9 to 9b1de85 Compare October 2, 2018 20:11

richardlau closed this Mar 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

esm: Implement esm mode flag #18392

esm: Implement esm mode flag #18392

guybedford commented Jan 26, 2018 •

edited

Loading

bmeck commented Jan 26, 2018 •

edited

Loading

jdalton Jan 26, 2018 •

edited

Loading

devsnek Feb 4, 2018

jdalton Feb 5, 2018

targos Feb 6, 2018

jdalton Jan 26, 2018 •

edited

Loading

jkrems Jan 26, 2018 •

edited

Loading

jdalton Jan 26, 2018 •

edited

Loading

MylesBorins Jan 27, 2018

jdalton commented Jan 26, 2018 •

edited

Loading

JacopKane commented Jan 26, 2018

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018 •

edited

Loading

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018 •

edited

Loading

jkrems commented Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

jkrems commented Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

giltayar commented Jan 26, 2018 •

edited

Loading

bmeck commented Jan 26, 2018

devsnek commented Apr 28, 2018 •

edited

Loading

ChALkeR commented Mar 1, 2019

devsnek commented Mar 1, 2019

richardlau commented Mar 19, 2019

GeoffreyBooth commented Mar 19, 2019

esm: Implement esm mode flag #18392

esm: Implement esm mode flag #18392

Conversation

guybedford commented Jan 26, 2018 • edited Loading

Checklist

Affected core subsystem(s)

bmeck commented Jan 26, 2018 • edited Loading

jdalton Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

devsnek Feb 4, 2018

Choose a reason for hiding this comment

jdalton Feb 5, 2018

Choose a reason for hiding this comment

targos Feb 6, 2018

Choose a reason for hiding this comment

jdalton Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

jkrems Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

jdalton Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

MylesBorins Jan 27, 2018

Choose a reason for hiding this comment

jdalton commented Jan 26, 2018 • edited Loading

JacopKane commented Jan 26, 2018

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018 • edited Loading

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018

bmeck commented Jan 26, 2018

jdalton commented Jan 26, 2018 • edited Loading

jkrems commented Jan 26, 2018 • edited Loading

jdalton commented Jan 26, 2018 • edited Loading

jkrems commented Jan 26, 2018 • edited Loading

jdalton commented Jan 26, 2018 • edited Loading

giltayar commented Jan 26, 2018 • edited Loading

bmeck commented Jan 26, 2018

devsnek commented Apr 28, 2018 • edited Loading

ChALkeR commented Mar 1, 2019

devsnek commented Mar 1, 2019

richardlau commented Mar 19, 2019

GeoffreyBooth commented Mar 19, 2019

guybedford commented Jan 26, 2018 •

edited

Loading

bmeck commented Jan 26, 2018 •

edited

Loading

jdalton Jan 26, 2018 •

edited

Loading

jdalton Jan 26, 2018 •

edited

Loading

jkrems Jan 26, 2018 •

edited

Loading

jdalton Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

jkrems commented Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

jkrems commented Jan 26, 2018 •

edited

Loading

jdalton commented Jan 26, 2018 •

edited

Loading

giltayar commented Jan 26, 2018 •

edited

Loading

devsnek commented Apr 28, 2018 •

edited

Loading