ARM64 support

[natan-abolafya]

Hi.

We're making an Windows ARM64 build for our electron application and native-reg is crashing with the following message:

No native build was found for platform=win32 arch=arm64 runtime=electron abi=106 uv=1 arm=8 libc=glibc node=16.14.2 electron=19.0.6

Is that a missing support on native-reg or am I missing something to make it work?

Thanks.

[simonbuchan]

Yeah, sorry, I'm prebuilding for x86/x64, but don't yet output ARM builds. Wouldn't be too hard to add, but I'm slightly concerned about bloating the package with binaries, though not enough to not build if there's no cross-build problems.

That said, prebuildify should be falling back to building locally if you have the toolchain setup, but I do remember issues with that kicking in when it shouldn't, perhaps I broke it. I'll take a look at that too.

[natan-abolafya]

Awesome, thanks for looking into it!

An alternative is perhaps to run the 32-bit binary instead of crashing since x86 emulation is in place on Windows 10+. But not sure how confusing it gets with the WowXXX stuff.

[simonbuchan]

Ok, so unfortunately or not, it seems node doesn't have official builds for ARM, which means I can't (easily at least) just add the pre-build for it (prebuildify links against the fetched node.lib).

As an alternative, I've restored the node-gyp fallback, give it a try in the 1.1.0-alpha.0 build I just published under the next tag - without an ARM machine I can't verify it, unfortunately.

[natan-abolafya]

this is great, thank you!

I will test this on Monday on a surface pro X and will update you.

[natan-abolafya]

hmm, unfortunately I can't build to test it yet. I'm not too familiar with gyp except remembering it being rather messy in the past :). So we always avoided it.

First problem is that all non-windows builds fail immediately but that can probably be addressed easily.
The second problem is cross-building electron to ARM64 fails. Pasting the error below. Let me know if this is due to something missing on my side or out of my control.

npm ERR! code 1
npm ERR! path C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron\dist\node_modules\native-reg
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node-gyp-build
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | win32 | x64
npm ERR! gyp info find Python using Python version 3.6.2 found at "C:\Users\admin\AppData\Local\Programs\Python\Python36-32\python.exe"
npm ERR! gyp http GET https://nodejs.org/download/release/v16.13.0/node-v16.13.0-headers.tar.gz
npm ERR! gyp http 200 https://nodejs.org/download/release/v16.13.0/node-v16.13.0-headers.tar.gz
npm ERR! gyp http GET https://nodejs.org/download/release/v16.13.0/SHASUMS256.txt
npm ERR! gyp http GET https://nodejs.org/download/release/v16.13.0/win-x86/node.lib
npm ERR! gyp http GET https://nodejs.org/download/release/v16.13.0/win-x64/node.lib
npm ERR! gyp http GET https://nodejs.org/download/release/v16.13.0/win-arm64/node.lib
npm ERR! gyp http 200 https://nodejs.org/download/release/v16.13.0/SHASUMS256.txt
npm ERR! gyp http 404 https://nodejs.org/download/release/v16.13.0/win-arm64/node.lib
npm ERR! gyp http 200 https://nodejs.org/download/release/v16.13.0/win-x86/node.lib
npm ERR! gyp http 200 https://nodejs.org/download/release/v16.13.0/win-x64/node.lib
npm ERR! gyp info find VS using VS2022 (17.2.32616.157) found at:
npm ERR! gyp info find VS "C:\Program Files\Microsoft Visual Studio\2022\Professional"
npm ERR! gyp info find VS run with --verbose for detailed information
npm ERR! gyp info spawn C:\Users\admin\AppData\Local\Programs\Python\Python36-32\python.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   'C:\\Users\\admin\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\gyp\\gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'msvs',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\BuildAgent\\work\\cabf95aa4bca1f9d\\src\\empower\\electron\\dist\\node_modules\\native-reg\\build\\config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\Users\\admin\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\WINDOWS\\system32\\config\\systemprofile\\AppData\\Local\\node-gyp\\Cache\\16.13.0\\include\\node\\common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=C:\\WINDOWS\\system32\\config\\systemprofile\\AppData\\Local\\node-gyp\\Cache\\16.13.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\admin\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=C:\\\\WINDOWS\\\\system32\\\\config\\\\systemprofile\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\16.13.0\\\\<(target_arch)\\\\node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=C:\\BuildAgent\\work\\cabf95aa4bca1f9d\\src\\empower\\electron\\dist\\node_modules\\native-reg',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'C:\\BuildAgent\\work\\cabf95aa4bca1f9d\\src\\empower\\electron\\dist\\node_modules\\native-reg\\build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp: C:\WINDOWS\system32\config\systemprofile\AppData\Local\node-gyp\Cache\16.13.0\include\node\common.gypi not found (cwd: C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron\dist\node_modules\native-reg) while reading includes of binding.gyp while trying to load binding.gyp
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onCpExit (C:\Users\admin\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:259:16)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:390:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
npm ERR! gyp ERR! System Windows_NT 10.0.19043
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\admin\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron\dist\node_modules\native-reg
npm ERR! gyp ERR! node -v v16.13.0
npm ERR! gyp ERR! node-gyp -v v8.4.1
npm ERR! gyp ERR! not ok
npm ERR! A complete log of this run can be found in:
npm ERR!     C:\WINDOWS\system32\config\systemprofile\AppData\Local\npm-cache\_logs\2022-07-16T15_06_41_187Z-debug-0.log
npm ERR! Lifecycle script `copy-assets` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: empower-electron 
npm ERR!   at location: C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron 
npm ERR! Lifecycle script `build` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: empower-electron 
npm version:
npm ERR!   at location: C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron 
Exiting script with reason: ''
npm ERR! Lifecycle script `pack-win` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: empower-electron 
npm ERR!   at location: C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron 
Error:  { code: 1 }

[simonbuchan]

Sorry for the late reply!

This one is weird, the relevant error is:

C:\WINDOWS\system32\config\systemprofile\AppData\Local\node-gyp\Cache\16.13.0\include\node\common.gypi not found (cwd: C:\BuildAgent\work\cabf95aa4bca1f9d\src\empower\electron\dist\node_modules\native-reg) while reading includes of binding.gyp while trying to load binding.gyp

Firstly, the C:\WINDOWS\system32\config\systemprofile path implies that the build is running under the SYSTEM account, which sounds... bad? But this does look like a build agent, so perhaps not too bad? But there is also '-Dnode_gyp_dir=C:\\Users\\admin\\AppData\\Roaming\\npm\\node_modules\\npm\\..., so it could be incorrect. Notably, that's %APPDATA%, as opposed to %LOCALAPPDATA%.

Secondly, assuming that is correct, it's very weird that ..\include\node\common.gypi wasn't found, That's fetched into that path above with the GET to https://nodejs.org/download/release/v16.13.0/node-v16.13.0-headers.tar.gz, which indeed does contain it:

C:\Users\simon>tar -tf node-v16.13.0-headers.tar.gz
node-v16.13.0/
node-v16.13.0/include/
node-v16.13.0/include/node/
node-v16.13.0/include/node/common.gypi
node-v16.13.0/include/node/config.gypi
... many many more

Note that it's not dependent on the target arch.

Thirdly, my builds with prebuildify on ARM get past that.... sigh.

So my best guess is that somehow a user switch or screwy environment variables are confusing node-gyp here?

But good-ish news, I did some dark deeds and got a prebuildify ARM64 build by fooling it into using the electron dist-url, which should be fine for node too, since this is a N-API addon. Should. Take a look at 1.1.0-alpha.1, again on the next tag.

[natan-abolafya]

yes, you're right, this is the build agent. It sounds sketchy but we need it as the SYSTEM to get certain access (though perhaps that could be done in other ways 🤔). But it's going to C:\\Users\\admin\\AppData\\ looks quite sketchy as you said. Maybe that's because how we updated npm on the agent was wrong and ended up putting things to admin user's %APPDATA%.

I probably should have tested this on my own machine also before jumping into the conclusion :).

It's great that you figured out the prebuilt! I have tried alpha.1 now. The builds all went fine, including the non-windows ones.
Tested it on the ARM64 device and it worked wonderfully.

[simonbuchan]

Sounds good, I'm going to let this sit for a week or so, just in case it starts eating babies or something, since I can't test it myself.

[natan-abolafya]

hehe ok. I won't be doing much testing on it yet though as there is quite a bit more to do on our ARM64 project before it's test ready.

but I'm happy to try specific things out on ARM64 if you need it.

[simonbuchan]

If it does anything without exploding I'd be very surprised if there's something wrong, but perhaps you could run the test npm script on ARM?

[natan-abolafya]

maybe a silly question but I can't seem to find the branch or tag for 1.1.0.

As my plan was to check that out and run npm run test on the repository. (And I was about to use an unofficial nodejs build for ARM64 :)).

Is there another way to do this that I don't know?

[simonbuchan]

I suppose that would help! I guess I'm used to our work tooling which pushes git as well as npm.

[simonbuchan]

Pushed to master.

[natan-abolafya]

well, that didn't go well. Setting up development environment is tricky on ARM64. I did install visual studio (with warnings and all) and stuff but node-gyp can't find the visual studio installation. Spent a good chunk of time to fix that but to no avail.

Again possibly newbie (a.k.a. full-stack engineer) question, is there any way to install the package via npm and run tests on it instead?

[simonbuchan]

Oh yes, I'd want to test with that anyway. You should just need to:

• make an new package somewhere,
• install mocha and native-reg,
• copy test/test.js to the new package (doesn't really matter where)
• edit the const reg = require(".."); line to be require("native-reg");
• Run mocha - simplest to copy the test script and run that I guess.

Shouldn't need to do this, to be clear, but if I had to guess you're installing visual studio 2022, and using an older version of node-gyp that doesn't know about it. If you want to try building, you will either need an updated node-gyp (which I think is not just npm -g i node-gyp? Might need to adjust path or something) or to use an older Visual Studio.

[natan-abolafya]

Done, 17 pass, no failure. With the unofficial ARM64 nodejs.

[simonbuchan]

Good to know! Thanks for taking the time to check.

[simonbuchan]

I assumed no news was good news, and this is now official as v1.1.0. Thank you for all you work and time!