Bundling and pkg fixes #26
Conversation
|
Ok, I have modified the build script. The typescript compile stage is now in the prebuild script and compiles to the
We can get a little more fancy with this. I think Next steps.
|
|
Fixed the worker script problem. I just added a new It's a little hacky, since the worker is loaded with We can do similar things for the .node files. For example the db loads them from the |
|
We'll worry about the Moving on to getting |
|
After reverting to node 18, running pkg is working. To move on from here I need to do a few things.
I think at this point I can start working on fixing |
|
What about node 19? |
|
It was easiest to revert the changes back to 18 for now. We can try version 19 but I think that's low priority. |
|
Staging fixes should be done here too. |
|
If we are using the esbuild output in the docker image build. This means our docker image no longer requires all of the npm packages that was structured before. This means all the relevant JS code has been embedded into the single The only thing that is required separately would be the worker script, and the This should mean the image becomes MUCH smaller. Especially if minification can work. If minification doesn't work atm, just create an issue for it, we can revisit after we have deployed testnet 6. |
|
A few things to consider here.
I think to ensure consistency - you'd want to incorporate esbuild into the nix scripts, not as part of |
|
Currently docker image doesn't really the esbuild output. It could continue using the regular dist. But only pkg needs the esbuild output. I think we can incorporate it directly into the Will need to think about how to incorporate the steps separately for docker later. |
|
I can have a look at this tomorrow. But please post up any build failure logs as to the reason for docker image build, if we can keep esbuild to the pkg build process, then docker image builds should be the same as before. |
|
Build is failing with the following. Doing prebuild normally runs fine. Not sure why it's an issue in Simplest fix would be to add the type annotations in but maybe not the best fix. |
|
Right now fixing up the tests and getting the build working can be done in parallel. It's possible to build and do a What's broken in tests right now is how errors are serialised and deseralized. It's something I'm working on int |
It's possible solving MatrixAI/Polykey#532 will have a side-effect of solving this problem. |
|
So you have 2 problems right now:
I can look at 2. now. |
| const { WebSocketClient } = await import('@matrixai/ws'); | ||
| const clientUtils = await import('polykey/dist/client/utils'); |
There was a problem hiding this comment.
Seems problematic...
Surely it's possible to expose the necessary abstractions directly from PK?
| webSocketClient = await WebSocketClient.createWebSocketClient({ | ||
| expectedNodeIds: [clientOptions.nodeId], | ||
| config: { | ||
| verifyPeer: true, | ||
| verifyCallback: async (certs) => { | ||
| await clientUtils.verifyServerCertificateChain( | ||
| [clientOptions.nodeId], | ||
| certs, | ||
| ); | ||
| }, | ||
| }, | ||
| host: clientOptions.clientHost, | ||
| port: clientOptions.clientPort, | ||
| logger: this.logger.getChild(WebSocketClient.name), | ||
| }); |
There was a problem hiding this comment.
Shouldn't PolykeyClient encapsulate the WS client setup. I don't like having every command setup its own websocket transport. That seems very incorrect.
There was a problem hiding this comment.
I think we wanted PolykeyClient to be agnostic to transport like the RPC?
There was a problem hiding this comment.
How can PK client be agnostic, and yet PK agent be not agnostic. Doesn't make sense.
There was a problem hiding this comment.
@amydevs can you help on this problem?
There was a problem hiding this comment.
Ok once WebSocketClient is moved into the PolykeyClient. That should mean @matrixai/ws should be removed from the package.json.
| streamFactory: (ctx) => webSocketClient.startConnection(ctx), | ||
| streamFactory: () => webSocketClient.connection.newStream(), |
There was a problem hiding this comment.
It does not make sense. PolykeyClient should be doing this all automatically.
|
To deal with docker build I need to try solve MatrixAI/Polykey#532, then you can try and see if that fixes it. |
|
I had to disable 2 tests for |
What does this have to do with RPC? |
|
It's the timer is being refreshed inside the RPC logic. It's being refreshed after timing out and thus throwing an error. |
|
I'm disabling |
Why is this failing? |
|
Ok, all tests are passing now except for those 3 tests I disabled. I need to write a test and fix for |
The test starts a polykey agent and runs a bootstrap concurrently. It expects that one of these will fail but both are succeeding. If I had to guess its bootstrapping and then starting without conflicting as a race condition. A failure of the test itself essentially. |
|
I pushed some fixes to |
|
Ok, after doing a prerelease on I'm guessing the last line before the problem happens is I recently removed the Also note that, If I switch back to linking the local |
CMCDragonkai
force-pushed
the
feature-bundling
branch
from
2d508d6 to
c5326ce
Compare
|
Task 3 will depend on fixing the test case. If @addievo can get the encapsulation fully working, we could incorporate here, but I think this will probably merge first, then another PR to integrate that change. |
npm issues that track the problem of missing |
|
Docker build works fine right now. |
|
So the esbuilt bundle seems to behave differently in relation to signal handling - in particular cleanup. The issue is that This may be related to the occasional exception I get when running polykey agent against an existing node state. Also I had seen this earlier: #26 (comment) but without the earlier |
|
@tegefaulkes you should update your VBox platform, the docker problem is an old problem we already know from a while ago. The solution is to match our other platform config. NixOS/nixpkgs#170279 |
|
Ok, so we have found two small issues with the CLI that I think are inter-related.
I'm looking into problem 1. right now. I think 2. is caused by 1. |
|
Ok this is updated with alpha.13 which uses However still need to fix the reported problems @tegefaulkes is on and the existing test problems here. Now I can try to get the CI ready. |
|
progress update. Problem 2 with the taskmanager. The error is coming out of the worker script. The data seems to be formatted badly in the esbuild build. worker function being called. decrypt(key, cipherText) {
console.log('key', key);
console.log('cipherText', cipherText);
const plainText = keysUtils.decryptWithKey(Buffer.from(key), Buffer.from(cipherText));
if (plainText != null) {
return (0, worker_1.Transfer)(plainText.buffer);
}
else {
return;
}
}normal build data esbuild data |
|
So the |
|
Ok all tests is passing now. But there's a weird exit gracefully problem: Possibly related to MatrixAI/js-timer#15, not a problem atm. |
|
@tegefaulkes can you start squashing this so it can be merged. We can merge in the replacement of websocketclient afterwards. Also make sure you're squashing any of my wip commits. |
CMCDragonkai
force-pushed
the
feature-bundling
branch
from
ace9535 to
f3f0ce5
Compare
* fixed parser imports to take it from the domain utilities, also exported the validate functions * all using `binParsers` now
* `esbuild` bundle is used for application build * changed to using `buildNpmPackage` instead of `node2nix` * changed to using `@yao-pkg/pkg` instead of `pkg` to use v18.15 nodejs
…es/CommandList.ts`
This makes sure the `esbuild` treats `threads` as an external dependency. There is some monkey patching going on that doesn't work if it's bundled.
It's a race condition with propagating keys changes. We needed to wait and then attempt calls with the new nodeId. I think there's a separate issue when verifying with the old nodeId. That needs to be checked inside `Polykey` though.
CMCDragonkai
force-pushed
the
feature-bundling
branch
from
f3f0ce5 to
331b9f8
Compare
|
Ok this is squashed, merging to staging to confirm the final CI steps is working which should also be auto-building and pushing to the ECR. |
|
However, still pending major issue is MatrixAI/Polykey#582. That's next priority for PK CLI. |
Description
This PR focueses on 3 things.
Issues Fixed
Tasks
esbuildto bundle thePolykey CLI.Polykey CLIis in a workable state.node2nixwithbuildNpmPackagevercel/pkgwith@yao-pkg/pkg[ ] 8. Client verification of server certificate needs to be more robust- to be resolved here: Encapsulate WebSocketClient in PolykeyClient Polykey#582 (comment)Final checklist