JS source should not be wrapped. #17396

hashseed · 2017-11-30T08:58:04Z

When Node.js executes JavaScript, it internally wraps them into an IIFE:

Module.wrap = function(script) {
  return Module.wrapper[0] + script + Module.wrapper[1];
};

Module.wrapper = [
  '(function (exports, require, module, __filename, __dirname) { ',
  '\n});'
];

There are some problems with this:

Invalid JavaScript is parsed as correct. This following example executes just fine:

});

(function() {
  console.log(1);
})();

(function() {

This wrapper shows up as part of source in the DevTools. Since the start line and column are passed as 0, the source positions of errors and break points at least for the first line of the script are off:

$ cat test.js
throw new Error();

$ node test.js
/usr/local/google/home/yangguo/node/test.js:1
(function (exports, require, module, __filename, __dirname) { throw new Error();
                                                              ^
Error
    at Object.<anonymous> (/usr/local/google/home/yangguo/node/test.js:1:69)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
    at Function.Module.runMain (module.js:609:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:578:3

This disparity between actual source and source being passed to V8 also affects code coverage. See https://bugs.chromium.org/p/v8/issues/detail?id=7119
Since Node.js reports 0 for start line and column, V8 has no way to adjust the source offsets reported for coverage.

The better way to do it might be to use ``v8::ScriptCompiler::CompileFunctionInContext`, where we ask V8 to wrap the source in a function context. See examples here.

cc @schuay @bcoe

The text was updated successfully, but these errors were encountered:

fhinkel · 2017-11-30T09:11:13Z

/cc @nodejs/v8

MylesBorins · 2017-11-30T09:21:02Z

@hashseed is it safe to assume that V8 function would allow us to continue to inject all the same lexically scoped variables?

Would there be a way to call this function via a macro in js land to avoid having to juggle things between the layers? Can you imagine any reason this would cause execution to be different? My biggest concern right now would be how this would affect current ecosystem tools that might be doing magic to account for the wrapper

My gut is that this may be semver major, and potentially break some stuff in the ecosystem... If my gut is wrong this could potentially be semver patch, which would be quite exciting

bnoordhuis · 2017-11-30T09:47:58Z

The better way to do it might be to use ``v8::ScriptCompiler::CompileFunctionInContext`, where we ask V8 to wrap the source in a function context.

I don't remember the details but it didn't quite work last time I tried it: #9273 (comment)

hashseed · 2017-11-30T12:44:59Z

Hehe... yeah now that I looked closer v8::ScriptCompiler::CompileFunctionInContext suffers from the same issue. It also just wraps the source at some point.

Looks like I'll have to fix this in V8 first.

@MylesBorins my hope is that it would semantically not break anything.

benjamingr · 2017-11-30T16:23:03Z

We need to be careful to not break existing code - returning in a module for instance. My gut feeling is that this breaks a lot of code in the wild doing something like:

if(!process.env.SOME_CONFIG) {
    module.exports = "bar";
    return;
}

causeEffects();
module.exports = "foo";

devsnek · 2017-11-30T16:31:32Z

i feel like we should treat that like we treat

});
some code
(function() {

and just consider it unsupported behavior, as users shouldn't depend on implementation details

isaacs · 2017-11-30T16:44:46Z

Breaking return in a commonjs module context will cause significant havoc.

A V8 api that preserves current behavior without an iife wrap would be nice, though. What about using new Function()? Does that have the same problems?

devsnek · 2017-11-30T16:52:34Z

wouldn't new Function() just become function anonymous(some args) { some body } and have the same problem?

benjamingr · 2017-11-30T17:32:31Z

From what I understand the suggested CompileNewFunctionInContext would still be a function and would work - I was pointing out return in case different solutions are suggested.

(idea: would dropping the \n in Module.wrapper fix the line numbering?)

isaacs · 2017-11-30T17:55:36Z

@devsnek I just tested, and yes, it does. What's worse, the offset may change between V8 versions, unlike the known iife wrapper Node uses.

@benjamingr I think the same problem applies to CompileNewFunctionInContext. It still adds wrapper code, but then it's wrapper code we can't safely account for.

My suggestion is to leave the CommonJS system frozen, and instead focus on making things right with the es-module system. Whatever shortcomings Node.js's CommonJS implementation has (and it has several), they haven't prevented a vibrant ecosystem from using them to build things, and fixing those shortcomings is almost certainly going to break that ecosystem in profound and hard-to-predict ways.

isaacs · 2017-11-30T17:57:35Z

(idea: would dropping the \n in Module.wrapper fix the line numbering?)

Only in cases where the error happens on the last line of the script because something wasn't terminated. For example:

$ cat end.js
if ('foo') {

$ node end.js
/Users/isaacs/dev/js/ayo/end.js:3
});
 ^

SyntaxError: Unexpected token )

But otherwise, the only issue is the column offset on line 1.

jdalton · 2017-11-30T18:15:13Z

⚠️ From a user-land view I want to be very careful about changes to Node CJS wrapping. If there is an API change (should try to avoid) it should still be user-land accessible since it does aid in dev related packages. ⚠️

Update:

Just noticed @isaacs' comment:

My suggestion is to leave the CommonJS system frozen, and instead focus on making things right with the es-module system. Whatever shortcomings Node.js's CommonJS implementation has (and it has several), they haven't prevented a vibrant ecosystem from using them to build things, and fixing those shortcomings is almost certainly going to break that ecosystem in profound and hard-to-predict ways.

💯 ++

devsnek · 2017-11-30T18:36:15Z

+1 to freezing commonjs system, anything to keep people moving to esm is good

bcoe · 2017-11-30T18:49:05Z

@jdalton @isaacs @hashseed locking the existing approach for CJS seems workable, as long as the new ESM system provides appropriate context in the scriptParsed event of the debugger. Here's what the meta information looks like currently when you parse a module in v8:

{
  "method": "Debugger.scriptParsed",
  "params": {
    "scriptId": "72",
    "url": "/Users/benjamincoe/bcoe/c8/test/fixtures/b.js",
    "startLine": 0,
    "startColumn": 0,
    "endLine": 12,
    "endColumn": 3,
    "executionContextId": 1,
    "hash": "1A47AD409B904C96B5BC49D6A58187D4B2D1C2C4",
    "isLiveEdit": false,
    "sourceMapURL": "",
    "hasSourceURL": false,
    "isModule": false,
    "length": 171,
    "stackTrace": {
      "callFrames": [
        {
          "functionName": "createScript",
          "scriptId": "41",
          "url": "vm.js",
          "lineNumber": 79,
          "columnNumber": 9
        }
      ]
    }
  }
}

My expectation would be that anything that comes through the new module system would have isModule = true and would not have wonky offsets?

Another thing that I think would be nice would be to expose the fact that we have a 62 byte prefix in a variable, e.g., require.cjsPrefix? This would allow tooling to avoid the 62 byte magic #.

jdalton · 2017-11-30T18:54:23Z

@bcoe

Another thing that I think would be nice would be to expose the fact that we have a 62 byte prefix in a variable, e.g., require.cjsPrefix?

Can be gotten from require("module").wrapper[0] or module.constructor.wrapper[0].

ofrobots · 2017-11-30T18:56:27Z

Or

export const MODULE_WRAP_PREFIX_LENGTH =
    require('module').wrap('☃').indexOf('☃');

from here.

bcoe · 2017-11-30T19:57:33Z

It looks like we are setting isModule to true 👍

https://github.com/nodejs/node/blob/master/src/module_wrap.cc#L105

So, I think the only work that would shake out of this is ensuring that ES modules don't have the same wrapper offset issue.

hashseed · 2017-11-30T20:18:04Z

I guess for the particular use case of code coverage we can get the wrapper header offset directly and subtract it from the coverage offsets. Might still be nice to move towards hiding this implementation detail though, without changing the general semantics of the function scope.

bmeck · 2017-11-30T21:51:25Z

We should probably check if any existing popular projects are using this: https://github.com/search?l=JavaScript&p=2&q=%22module.wrapper%22+%22require%28.module.%29%22&type=Code&utf8=%E2%9C%93 , a quick look didn't show anything terrible

jdalton · 2017-11-30T22:03:53Z

a quick look didn't show anything terrible

It shows usage.
(I've used it on more than one occasion too.)

What's wrong with @isaacs' stance?

bmeck · 2017-11-30T22:07:06Z

Nothing, I am not +/- to it, just interested that no statistics were being brought to the table. None of the uses seemed to be abusing the feature for Ill gotten gains while also being popular enough that we need to replicate such a feature for ESM.

jdalton · 2017-11-30T22:11:12Z

Okay. FWIW I don't think anyone was suggesting the CJS wrapper APIs be added to ESM, just suggesting they not be removed from CJS if it can be avoided.

schuay · 2017-12-01T08:49:20Z

I think the same problem applies to CompileNewFunctionInContext. It still adds wrapper code, but then it's wrapper code we can't safely account for.

Currently, yes. But we're discussing possible solutions where the source string would remain unmodified and the wrapper would be created only as AST nodes by the parser. That'd avoid any complications around weird line/column offsets, and we'd have better verification that the given script is actually valid JS.

bnoordhuis · 2017-12-01T09:39:38Z

^ That seems strictly superior to me to what we currently have. I'm +1 on switching over.

jdalton · 2017-12-01T15:52:26Z

@schuay

Currently, yes. But we're discussing possible solutions where the source string would remain unmodified and the wrapper would be created only as AST nodes by the parser. That'd avoid any complications around weird line/column offsets, and we'd have better verification that the given script is actually valid JS.

That's pretty cool. It looks like something that could be exposed by vm and used by Module._compile without changing the surrounding APIs (Module.wrap, Module.wrapper, etc.) which is great!

bmeck · 2017-12-01T16:04:04Z

@jdalton I don't think this would be passed the entire function string, just the body of the function. Which wouldn't be able to be compatible with .wrap/.wrapper since it would return a function instead of a string and would not be accepting the function prefix/suffix.

jdalton · 2017-12-01T16:19:42Z

@bmeck

I don't think this would be passed the entire function string, just the body of the function. Which wouldn't be able to be compatible with .wrap/.wrapper

Oh, _{ok never mind then} 😢

Exposing it to vm would still be rad on its own though (but that's kinda off topic)

hashseed · 2017-12-18T10:24:42Z

I landed the patch to v8::ScriptCompiler::CompileFunctionInContext upstream: https://chromium.googlesource.com/v8/v8.git/+/1586f37f2d1f06415dbaf0c03b0f6e0702450ce9

hashseed · 2017-12-18T10:27:03Z

@bnoordhuis the stack trace line/column issue should be fixed now. The current API does not support code caching, but it would still be interesting to see whether it works for require. It should be already be a bit faster than using wrappers, and I have some ideas to make it faster still.

hashseed · 2018-06-20T09:34:58Z

In reply to concerns above: return still works.

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: nodejs#17396

PR-URL: #21573 Fixes: #17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: #17396 PR-URL: #21573 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

PR-URL: #21573 Fixes: #17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: #17396 PR-URL: #21573 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

PR-URL: #21573 Fixes: #17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

PR-URL: nodejs#21573 Fixes: nodejs#17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: #17396 Backport-PR-URL: #27124 PR-URL: #21573 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Backport-PR-URL: #27124 PR-URL: #21573 Fixes: #17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: #17396 Backport-PR-URL: #27124 PR-URL: #21573 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Backport-PR-URL: #27124 PR-URL: #21573 Fixes: #17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: #17396 Backport-PR-URL: #27124 PR-URL: #21573 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Backport-PR-URL: #27124 PR-URL: #21573 Fixes: #17396 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Use vm.compileFunction (which is a binding for v8::CompileFunctionInContext) instead of Module.wrap internally in Module._compile for the cjs loader. Fixes: nodejs/node#17396 PR-URL: nodejs/node#21573 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

mscdex added the module Issues and PRs related to the module subsystem. label Nov 30, 2017

This was referenced Nov 30, 2017

proposal: standardize header size prepended before executing scripts in v8 #17256

Closed

native v8 test coverage - remaining blockers #17337

Closed

bnoordhuis mentioned this issue May 28, 2018

Error stack gives incorrect column number #2860

Closed

ryzokuken mentioned this issue Jun 28, 2018

module: use compileFunction over Module.wrap #21573

Closed

4 tasks

demurgos mentioned this issue Dec 21, 2018

feat: allow script wrapper length to be specified bcoe/c8#51

Merged

2 tasks

ryzokuken closed this as completed in d345b0d Feb 19, 2019

This was referenced Mar 7, 2019

Source maps are misaligned GoogleChromeLabs/ndb#243

Closed

adjust source maps to account for node script wrapping GoogleChromeLabs/ndb#244

Merged

SimenB mentioned this issue Dec 1, 2019

feat: add support for vm.compileFunction jsdom/jsdom#2731

Closed

Melab mentioned this issue Jul 23, 2022

CommonJS module loader is error-prone oracle/graaljs#621

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JS source should not be wrapped. #17396

JS source should not be wrapped. #17396

hashseed commented Nov 30, 2017

fhinkel commented Nov 30, 2017

MylesBorins commented Nov 30, 2017

bnoordhuis commented Nov 30, 2017

hashseed commented Nov 30, 2017

benjamingr commented Nov 30, 2017

devsnek commented Nov 30, 2017 •

edited

Loading

isaacs commented Nov 30, 2017

devsnek commented Nov 30, 2017 •

edited

Loading

benjamingr commented Nov 30, 2017

isaacs commented Nov 30, 2017

isaacs commented Nov 30, 2017

jdalton commented Nov 30, 2017 •

edited

Loading

devsnek commented Nov 30, 2017 •

edited

Loading

bcoe commented Nov 30, 2017

jdalton commented Nov 30, 2017 •

edited

Loading

ofrobots commented Nov 30, 2017

bcoe commented Nov 30, 2017

hashseed commented Nov 30, 2017

bmeck commented Nov 30, 2017

jdalton commented Nov 30, 2017 •

edited

Loading

bmeck commented Nov 30, 2017

jdalton commented Nov 30, 2017 •

edited

Loading

schuay commented Dec 1, 2017

bnoordhuis commented Dec 1, 2017

jdalton commented Dec 1, 2017 •

edited

Loading

bmeck commented Dec 1, 2017

jdalton commented Dec 1, 2017 •

edited

Loading

hashseed commented Dec 18, 2017

hashseed commented Dec 18, 2017

hashseed commented Jun 20, 2018

JS source should not be wrapped. #17396

JS source should not be wrapped. #17396

Comments

hashseed commented Nov 30, 2017

fhinkel commented Nov 30, 2017

MylesBorins commented Nov 30, 2017

bnoordhuis commented Nov 30, 2017

hashseed commented Nov 30, 2017

benjamingr commented Nov 30, 2017

devsnek commented Nov 30, 2017 • edited Loading

isaacs commented Nov 30, 2017

devsnek commented Nov 30, 2017 • edited Loading

benjamingr commented Nov 30, 2017

isaacs commented Nov 30, 2017

isaacs commented Nov 30, 2017

jdalton commented Nov 30, 2017 • edited Loading

devsnek commented Nov 30, 2017 • edited Loading

bcoe commented Nov 30, 2017

jdalton commented Nov 30, 2017 • edited Loading

ofrobots commented Nov 30, 2017

bcoe commented Nov 30, 2017

hashseed commented Nov 30, 2017

bmeck commented Nov 30, 2017

jdalton commented Nov 30, 2017 • edited Loading

bmeck commented Nov 30, 2017

jdalton commented Nov 30, 2017 • edited Loading

schuay commented Dec 1, 2017

bnoordhuis commented Dec 1, 2017

jdalton commented Dec 1, 2017 • edited Loading

bmeck commented Dec 1, 2017

jdalton commented Dec 1, 2017 • edited Loading

hashseed commented Dec 18, 2017

hashseed commented Dec 18, 2017

hashseed commented Jun 20, 2018

devsnek commented Nov 30, 2017 •

edited

Loading

devsnek commented Nov 30, 2017 •

edited

Loading

jdalton commented Nov 30, 2017 •

edited

Loading

devsnek commented Nov 30, 2017 •

edited

Loading

jdalton commented Nov 30, 2017 •

edited

Loading

jdalton commented Nov 30, 2017 •

edited

Loading

jdalton commented Nov 30, 2017 •

edited

Loading

jdalton commented Dec 1, 2017 •

edited

Loading

jdalton commented Dec 1, 2017 •

edited

Loading