esm: add experimental support for addon modules #55844

legendecas · 2024-11-14T01:18:25Z

Add experimental support to loading .node extension modules in ESM.

An addon exports two names default and module.exports, as same as import(cjs) where
its export names can not be preparsed. Addon export names can not be inferred until it is evaluated.

Fixes: #40541
Fixes: #55821

nodejs-github-bot · 2024-11-14T01:18:30Z

Review requested:

@nodejs/gyp
@nodejs/loaders

aduh95 · 2024-11-14T11:38:43Z

The commit message does not meet our guidelines, the word after the subsystem should be an infinitive verb. I suggest esm: add experimental support for addon modules

lib/internal/modules/esm/translators.js

test/addons/esm/test.js

doc/api/cli.md

lib/internal/modules/esm/translators.js

lib/internal/validators.js

src/node_options.cc

codecov · 2024-11-27T17:03:20Z

Codecov Report

Attention: Patch coverage is 86.73469% with 13 lines in your changes missing coverage. Please review.

Project coverage is 88.54%. Comparing base (a50f3d5) to head (d70a87c).
Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/modules/esm/translators.js	85.22%	13 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #55844      +/-   ##
==========================================
- Coverage   88.55%   88.54%   -0.01%     
==========================================
  Files         657      657              
  Lines      190243   190372     +129     
  Branches    36536    36546      +10     
==========================================
+ Hits       168461   168571     +110     
- Misses      14963    14991      +28     
+ Partials     6819     6810       -9

Files with missing lines	Coverage Δ
lib/internal/modules/esm/formats.js	`98.64% <100.00%> (+0.09%)`	⬆️
lib/internal/modules/esm/load.js	`91.94% <100.00%> (+0.10%)`	⬆️
src/node_options.cc	`88.00% <100.00%> (+0.01%)`	⬆️
src/node_options.h	`98.30% <100.00%> (+<0.01%)`	⬆️
lib/internal/modules/esm/translators.js	`91.50% <85.22%> (-1.44%)`	⬇️

... and 44 files with indirect coverage changes

nodejs-github-bot · 2024-11-28T13:40:12Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63746/

JakobJingleheimer

Thanks for this 🙂

for the tests, describe etc provide significantly better DX when tests fail

lib/internal/modules/esm/translators.js

JakobJingleheimer · 2024-11-30T10:44:09Z

test/addons/esm/test-esm.mjs

+
+export async function run() {
+  // binding.node
+  {


Please use describe + it

I personally find describe + it to be worse UX due to the reasons stated in #56027 (comment), unless there is a way to silence the passing tests and make the console logs from the failed test cases appear at the end together with the errors..

joyeecheung

Some suggestions, but non-blocking.

Addon export names can not be inferred until it is evaluated.

I wonder if there's anything we can do about it...potentially, defining some macro in node.h that the addons can use to export names with a specific pattern, which we can scan for by parsing the dynamic library (ideally through some new API in libuv, though we are already parsing binaries using postject for SEA which implements the parsing by itself...). Can be done as a follow-up, of course..

joyeecheung · 2024-12-02T15:06:56Z

lib/internal/modules/esm/formats.js

@@ -23,6 +24,10 @@ if (experimentalWasmModules) {
  extensionFormatMap['.wasm'] = 'wasm';
 }

+if (experimentalAddonModules) {
+  extensionFormatMap['.node'] = 'addon';


I am somewhat surprised to find that there is not a test for the node-addons exports condition...it seems the the files pointed to by node-addons won't get interpreted as addons unless they have the .node extension, though this issue predates this PR and also happens to require().

Taking a step back, I'm wondering if the format name could be renamed as node-addon/node-addons for alignment.

Sure, also I am not even sure if anyone is actually using it, it seems only useful to distinguish "if the current run times can load Node.js style addons" as it currently stands, so may not really be that big of a deal anyway....

I prefer the more concise addon

Actually I think I misread #55844 (comment) - I think for the format name, addon or node-addon would make more sense. node-addons would be a bit weird in the hooks since it's plural.

This loader format enum is specific for Node.js so I'll stick with the concise addon.

In any case, the exports condition question is separate - it's more about when a module that doesn't end with .node is pointed to by node-addons, should it be interpreted(translated) into an addon or should it be interpreted based on whatever extension it has - and if it has a different extension (e.g. dylib or dll), what should happen then. As I said in the first comment, it's fine to keep it as is since CJS also only considers .node, but it is a bit strange for the exports condition.

The reality is that the conditional exported entries are been loaded based on their extensions, not their entry types. With require(esm), it is possible to do the following already 👀:

{ "exports": { "import": "./index-require.cjs", "require": "./index-module.mjs" } }

lib/internal/modules/esm/translators.js

joyeecheung · 2024-12-02T15:18:20Z

test/addons/esm/test-esm.mjs

+
+export async function run() {
+  // binding.node
+  {


I personally find describe + it to be worse UX due to the reasons stated in #56027 (comment), unless there is a way to silence the passing tests and make the console logs from the failed test cases appear at the end together with the errors..

test/addons/esm/test-esm.mjs

test/addons/esm/test-import.js

joyeecheung · 2024-12-12T14:44:36Z

This needs a rebase.

nodejs-github-bot · 2024-12-16T15:07:39Z

CI: https://ci.nodejs.org/job/node-test-pull-request/64070/

doc/api/module.md

nodejs-github-bot · 2024-12-17T14:46:26Z

CI: https://ci.nodejs.org/job/node-test-pull-request/64085/

Co-authored-by: Antoine du Hamel <[email protected]>

nodejs-github-bot · 2024-12-18T00:25:23Z

CI: https://ci.nodejs.org/job/node-test-pull-request/64094/

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Nov 14, 2024

legendecas force-pushed the esm/addon branch from d4b9a9a to b384e80 Compare November 14, 2024 01:21

guybedford approved these changes Nov 14, 2024

View reviewed changes

legendecas force-pushed the esm/addon branch 3 times, most recently from c6ce9fd to 0c724cd Compare November 14, 2024 10:07

aduh95 reviewed Nov 14, 2024

View reviewed changes

lib/internal/modules/esm/translators.js Outdated Show resolved Hide resolved

aduh95 reviewed Nov 14, 2024

View reviewed changes

test/addons/esm/test.js Outdated Show resolved Hide resolved

legendecas changed the title ~~esm: experimental addon modules~~ esm: add experimental addon modules Nov 14, 2024

legendecas changed the title ~~esm: add experimental addon modules~~ esm: add experimental support for addon modules Nov 14, 2024

GeoffreyBooth reviewed Nov 14, 2024

View reviewed changes

legendecas force-pushed the esm/addon branch 2 times, most recently from 9841645 to 829cfc6 Compare November 27, 2024 14:42

legendecas marked this pull request as ready for review November 27, 2024 14:43

legendecas marked this pull request as draft November 27, 2024 14:54

legendecas force-pushed the esm/addon branch from 829cfc6 to 2ea455d Compare November 27, 2024 15:00

legendecas marked this pull request as ready for review November 27, 2024 15:03

GeoffreyBooth approved these changes Nov 27, 2024

View reviewed changes

legendecas added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 28, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 28, 2024

JakobJingleheimer reviewed Nov 30, 2024

View reviewed changes

joyeecheung approved these changes Dec 2, 2024

View reviewed changes

legendecas force-pushed the esm/addon branch from 389b5c5 to 7375fa0 Compare December 6, 2024 14:50

legendecas force-pushed the esm/addon branch from 7375fa0 to 16777e3 Compare December 16, 2024 14:23

legendecas added request-ci Add this label to start a Jenkins CI on a PR. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Dec 16, 2024

This comment was marked as outdated.

Sign in to view

esm: add experimental support for addon modules

1992c20

legendecas force-pushed the esm/addon branch from 16777e3 to 1992c20 Compare December 16, 2024 14:37

legendecas added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 16, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 16, 2024

aduh95 reviewed Dec 17, 2024

View reviewed changes

doc/api/module.md Outdated Show resolved Hide resolved

legendecas added 2 commits December 17, 2024 13:57

fixup! windows test failure

55efd59

fixup! use autolink in docs

cd68821

legendecas added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 17, 2024

aduh95 reviewed Dec 17, 2024

View reviewed changes

doc/api/module.md Outdated Show resolved Hide resolved

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 17, 2024

joyeecheung approved these changes Dec 17, 2024

View reviewed changes

fixup!

d70a87c

Co-authored-by: Antoine du Hamel <[email protected]>

legendecas added commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. request-ci Add this label to start a Jenkins CI on a PR. labels Dec 18, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

esm: add experimental support for addon modules #55844

esm: add experimental support for addon modules #55844

legendecas commented Nov 14, 2024

nodejs-github-bot commented Nov 14, 2024

aduh95 commented Nov 14, 2024

codecov bot commented Nov 27, 2024 •

edited

Loading

nodejs-github-bot commented Nov 28, 2024

JakobJingleheimer left a comment

JakobJingleheimer Nov 30, 2024

joyeecheung Dec 2, 2024

joyeecheung left a comment

joyeecheung Dec 2, 2024

legendecas Dec 5, 2024 •

edited

Loading

joyeecheung Dec 5, 2024

JakobJingleheimer Dec 7, 2024

joyeecheung Dec 12, 2024

legendecas Dec 16, 2024 •

edited

Loading

joyeecheung Dec 16, 2024 •

edited

Loading

legendecas Dec 16, 2024 •

edited

Loading

joyeecheung Dec 2, 2024

joyeecheung commented Dec 12, 2024

This comment was marked as outdated.

nodejs-github-bot commented Dec 16, 2024

nodejs-github-bot commented Dec 17, 2024

nodejs-github-bot commented Dec 18, 2024

esm: add experimental support for addon modules #55844

Are you sure you want to change the base?

esm: add experimental support for addon modules #55844

Conversation

legendecas commented Nov 14, 2024

nodejs-github-bot commented Nov 14, 2024

aduh95 commented Nov 14, 2024

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

nodejs-github-bot commented Nov 28, 2024

JakobJingleheimer left a comment

Choose a reason for hiding this comment

JakobJingleheimer Nov 30, 2024

Choose a reason for hiding this comment

joyeecheung Dec 2, 2024

Choose a reason for hiding this comment

joyeecheung left a comment

Choose a reason for hiding this comment

joyeecheung Dec 2, 2024

Choose a reason for hiding this comment

legendecas Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

joyeecheung Dec 5, 2024

Choose a reason for hiding this comment

JakobJingleheimer Dec 7, 2024

Choose a reason for hiding this comment

joyeecheung Dec 12, 2024

Choose a reason for hiding this comment

legendecas Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

joyeecheung Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

legendecas Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

joyeecheung Dec 2, 2024

Choose a reason for hiding this comment

joyeecheung commented Dec 12, 2024

This comment was marked as outdated.

nodejs-github-bot commented Dec 16, 2024

nodejs-github-bot commented Dec 17, 2024

nodejs-github-bot commented Dec 18, 2024

codecov bot commented Nov 27, 2024 •

edited

Loading

legendecas Dec 5, 2024 •

edited

Loading

legendecas Dec 16, 2024 •

edited

Loading

joyeecheung Dec 16, 2024 •

edited

Loading

legendecas Dec 16, 2024 •

edited

Loading