Support ng generate webWorker for libraries #15059
Comments
|
hey do any have any workaround ? |
|
Hey @klausj, It's been a while since you opened this issue, do you have any kind of solution / workaround on how to use a web worker in an Angular library? Thanks |
|
Hi @SansDK As a workaround I create an object URL which contains the code of a TypeScript method for the worker. The worker code is wrapped by a self executing function to be available to the worker. // Helper class to build Worker Object URL
export class WorkerHelper {
static buildWorkerBlobURL(workerFct: Function): string {
// Update 2020-07-16: Try to mitigate XSS attacks
// Should ensure a type check at runtime and reject an injected XSS string.
if(! (workerFct instanceof Function)) {
throw new Error(
'Parameter workerFct is not a function! (XSS attack?).'
)
}
let woFctNm = workerFct.name;
let woFctStr = workerFct.toString();
// Make sure code starts with "function()"
// Chrome, Firefox: "[wofctNm](){...}", Safari: "function [wofctNm](){...}"
// we need an anonymous function: "function() {...}"
let piWoFctStr = woFctStr.replace(/^function +/, '');
// Convert to anonymous function
let anonWoFctStr = piWoFctStr.replace(woFctNm + '()', 'function()')
// Self executing
let ws = '(' + anonWoFctStr + ')();'
// Build the worker blob
let wb = new Blob([ws], {type: 'text/javascript'});
let workerBlobUrl=window.URL.createObjectURL(wb);
return workerBlobUrl;
}
}A TypeScript class which requires a worker looks like this: export class ClassWithWorker{
private workerURL: string;
private worker: Worker | null;
constructor() {
this.worker = null;
this.workerURL = WorkerHelper.buildWorkerBlobURL(this.workerFunction)
}
/*
* Method used as worker code.
*/
workerFunction() {
self.onmessage = function (msg) {
let someInputData = msg.data.someInput;
// Work on someInput
...
// Post result
postMessage({someOutput:someData });
}
}
start(){
this.worker = new Worker(this.workerURL);
this.worker.onmessage = (me) => {
let result = me.data.someOutput;
...
}
this.worker.postmessage({someInput: someInputData});
}
stop() {
this.worker.terminate();
}
}The approach works but has some drawbacks:
|
|
Thanks for the explanation and the example, very useful! |
|
Just gonna drop my 2 cents, but allowing libraries to encapsulate web worker functionality away from consuming applications would be an absolute game changer. With this, a future where we'll only be chaining awaits vs handling call backs and subscriptions isn't far off. I suspect performance would skyrocket as library maintainers would be able to offload computations completely from the UI thread. Code quality would increase in the ecosystem as a whole. |
dgp1130
added
area: @schematics/angular
freq1: low
|
Talked about this a bit in amongst the team. Workers have a lot of use cases for libraries. The challenge here is that workers must be loaded from a URL, and a library typically can't assume anything about the URL layout of a particular application. Fortunately this is pretty easy to work around. Libraries can just export a function that does what they want and applications can easily import and call this function from their own worker. For example: // my-lib.ts
export function work() {
console.log('Does work');
}Then the application can define a worker file which calls this library: // my-worker.worker.ts
import { work } from 'my-lib';
work();The worker can be started like any other by the application: // app.component.ts
const worker = new Worker('./my-worker.worker');The We're curious to hear if there are other use cases which might not be well-supported by this kind of design. The only one I can think of would be a library which wants to manage its workers directly (like a The workaround previously mentioned is quite clever but not usable for production.
|
dgp1130
removed
the
needs: discussion
|
That sounds nice, but ideally, libraries would automatically wire up in a consuming application. If there were some kind of standard that we developed for workers inside a library, the compiler would be able to automatically wire up service workers correctly, would they not? This same issue somewhat exists for assets, styles, and scripts. In order to consume some libraries, you have to modify the consuming applications angular.json sections manually. To circumnavigate this, I've seen libraries create a service that during App Initialization will essentially wire up scripts by appending to document body some CDN paths. In our case, we end up deploying npm-install scripts in the packages that has to search, then modify, these sections of the angular application. While scaffolding exists, its extremely heavy duty for what is essentially a json merge of these sections from library and application. |
|
Here's a use case to mention: I have an angular library for websocket comm. It contains an Angular Service, Sometimes, messages get very big (~300MB) and this causes @dgp1130 thank you for posting such detailed instructions. I followed your approach:
Here are the drawbacks: If I simply import the code directly from its file (rather than the barrel), the error goes away. TLDR - it does indeed seem like there are ways of work around this limitation, but it results in duplicate code across applications, and there are all sorts of nuiances that I'm still discovering. I somewhat understand the challenges you describe here, but it would certainly be nice if I could simply package my web worker into a library that any application could use. |
@dgp1130 thanks for your comment. |
|
@klausj, I don't immediately see anything exploitable in your https://stackblitz.com/edit/typescript-b4g24s // Vulnerable to XSS, do NOT use this example!
class VulnerableWorkerWrapper {
static worker?: Worker;
work() {
self.onmessage = ({ data }) => {
console.log(`${data.key}: ${data.value}`);
};
}
start() {
VulnerableWorkerWrapper.worker = new Worker(WorkerHelper.buildWorkerBlobURL(this.work));
for (const [ key, value ] of Object.entries(this)) {
VulnerableWorkerWrapper.worker.postMessage({ key, value });
}
}
};
const vulnerableWorker = new VulnerableWorkerWrapper();
const params = new URL(window.location.href).searchParams;
for (const [ key, value ] of Array.from(params.entries())) {
vulnerableWorker[key] = value; // BAD!!! What happens if `key === 'work'`?
}
vulnerableWorker.start();This example can be trivially exploited by setting JavaScript code in the Ultimately |
|
Thanks for your interesting demo! And I updated my code snippet with a type check which should ensure that the builder method only accepts values of type 'Function'. Injected code strings should be rejected with an error at runtime. |
|
@dgp1130 Sanitizing input is always a must, even in in Angular today. If Angular allows interpolation without requiring the user to sanitize, I don't see why this case should be any different. The power that could be unleashed by allowing libraries to encapsulate and provide worker support is just too tempting. Like i mentioned before, we already hook into npm install in order to auto wire up asset, style, and dependent package support. I'm beginning to do the same with this just for investigation. |
|
Using some of @dgp1130 thoughts, I was able to kludge together a solution that worked. Not ideal. Won't be putting this into production in any way shape or form until there's less glue required to get this to work. Getting the thing to compile required absolute paths in any of the webworker code. Got a bunch of typing errors when any library was references by relative or module name. It seems when writing the code in a library, there can be no javascript imports or else it won't compile correctly. On top of that, in the parent application, I had to reference the js file directly in the web worker (after compiling the worker using tsc manually in the library and deploying it with the package).
forget about doing that. You'll be met with typescript typings mismatches between the dom library and the worker library. It also seems that any functionality in the lib must be restricted to exporting functions. I originally wrote my library code in a class. In the parent worker, using new MyClass(). Instantly would break. The component then requires an input of type Worker that the parent passes to it (since calling new Worker inside the library doesn't work at all). That being said, the performance improvement, even without using Transferable between the worker and parent, was astounding. Better support/encapsulation of this technology in angular would be revolutionary. |
|
I am also trying to do the same thing. So far the only way I can make it work is by:
I have also tried to keep the logic for the web worker in the library as much as possible, by keeping the initialization of the worker object in the library. But to do this, I have to pass the hard-coded file path to the worker script. Passing the path by using Input or InjectionToken does not work. Another workaround I have tried is to write the worker script as a JS script, then import it through angular.json. But not able to make this work. It gives me a mimeType (text/html) not matched. Has anyone tried the 3rd workaround? |
|
I will leave a note here, I need this too! I am clustering 62000 markers on a map. When I do it from the main thread the process takes up to several seconds on a normal PC... I would love to be able to split it. I have the splitting ready, I only need to be able to create a worker... |
|
While I hope that at some point we'll have proper support for this, it's not too painful to go around this limitation by using dependency injection. The only thing that is blocking us to use a webworker in a library is the creation of the worker itself, which is pretty much one line: Disclaimer: I'm only sharing a solution which @zakhenry introduced within our project a long time ago so most of the credits here go to him 😸! Let see how to achieve this using DI. I've created a small open source repro here: https://github.com/maxime1992/demo-webworker-library The main idea being:
Here's a schema representing the overall idea of what we want to simulate through DI:
First of all, I've picked a slow implementation for the fibonacci function (on purpose): projects/fibonacci/src/lib/fibonacci.ts // I've chosen a slow implementation of fibonacci on purpose
// so that it takes more time to run otherwise an optimised
// version could take less than 1s to compute fibonacci(3000)
// while this one will struggle with fibonacci(30)
// https://stackoverflow.com/questions/11287418/why-is-this-js-code-so-slow
export const fibonacci = (n) => {
if (n < 2) {
return n;
} else {
return fibonacci(n - 1) + fibonacci(n - 2);
}
};Then we can build the library **projects/fibonacci-webworker/src/lib/fibonacci.ts**
/// <reference lib="webworker" />
// this file is excluded from the compilation of the `fibonacci-webworker` library
// this is due to the fact that an app has to import it directly (instead of a path
// coming from a ts alias path)
import { fibonacci } from 'fibonacci';
addEventListener('message', ({ data }) => {
postMessage(fibonacci(data));
});Important notes:
We can now create a token which we'll use later on to provide our web worker through DI: projects/fibonacci-webworker/src/lib/fibonacci.token.ts import { InjectionToken } from '@angular/core';
export const FIBONACCI_WEBWORKER_FACTORY = new InjectionToken<() => Worker>(
'fibonacci'
);As you can see it's been defined as a factory so that we can easily create new instances of the worker on demand 👍. Then we expose this token by exporting it from Now, within our component in projects/demo-lib-consuming-webworker/src/lib/demo-lib-consuming-webworker.component.ts import { Component, Inject } from '@angular/core';
import { FIBONACCI_WEBWORKER_FACTORY } from 'fibonacci-webworker';
import { interval } from 'rxjs';
@Component({
selector: 'lib-demo-lib-consuming-webworker',
template: `
<p>
Timer just to prove that the page isn't frozen while we compute fibonacci
(which it would if we were doing it on the main thread!):
{{ timer$ | async }}
</p>
<div *ngIf="result; else computingTpl">
<h1>Fibonacci 45</h1>
<p>{{ result | json }}</p>
</div>
<ng-template #computingTpl>
Computing fibonacci(45)... Be patient =)!
</ng-template>
`,
})
export class DemoLibConsumingWebworkerComponent {
public result;
public timer$ = interval(1000);
constructor(
@Inject(FIBONACCI_WEBWORKER_FACTORY) fibonacciWebworkerFactory: () => Worker
) {
const fibonacciWebworker = fibonacciWebworkerFactory();
fibonacciWebworker.onmessage = ({ data }) => {
this.result = data;
};
fibonacciWebworker.postMessage(45);
}
}Last remaining bit to have everything working is to define the worker factory from the main app and use the In order to include the web worker within the compilation of the app, we add a {
"extends": "./tsconfig.json",
"compilerOptions": {
"outDir": "./out-tsc/worker",
"lib": ["es2018", "webworker"],
"types": []
},
"include": [
"src/**/*.worker.ts",
"projects/fibonacci-webworker/src/lib/fibonacci.ts"
]
}Note that the most important line is within the In order to use it we've got to declare it into
And the line we want to add is:
Finally, our import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { RouterModule } from '@angular/router';
import { FIBONACCI_WEBWORKER_FACTORY } from 'fibonacci-webworker';
import { AppComponent } from './app.component';
@NgModule({
declarations: [AppComponent],
imports: [
BrowserModule,
RouterModule.forRoot([
{
path: '',
loadChildren: () =>
import('demo-lib-consuming-webworker').then(
(m) => m.DemoLibConsumingWebworkerModule
),
},
]),
],
providers: [
{
provide: FIBONACCI_WEBWORKER_FACTORY,
useValue: function (): Worker {
return new Worker('projects/fibonacci-webworker/src/lib/fibonacci', {
name: 'fibonacci.worker',
type: 'module',
});
},
},
],
bootstrap: [AppComponent],
})
export class AppModule {}
And here's the result of the app:
Important note:
While I'm aware the above may seem a bit scarry and long, I've been giving quite a lot of details and the number of changes to actually build all of this (not including the creation of the 3 different libs because it's all done by the CLI and we don't have to care about it) is not that big:
I reckon following this would take the first time maybe... 5 to 10mn and reusing the worker within other bits of the same app once already created/declared in the app would take 10s 😺!
|
|
@maxime1992 Thank you so much for that detailed writeup, it was incredibly helpful and seems like a good interim solution. Notable downsides that I'd look for a native angular feature to solve:
Again, thanks for your solution, I'll be using that for now and looking for a solution from Angular in the long-term. |
|
are there any chances it will be implemented in the near future? |
|
We tried to play around a bit with the different approaches mentioned here, but in the end it always comes down to the problem that web-worker TypeScript files aren't compiled towards JavaScript during The approach of @maxime1992 consisted of a lot of "indirections" to make web-workers actually work, but the approach is useless if you're not inside the sphere of a shared monorepo. I don't see it working as soon as library and application do live in separate repositories, because the following assumption wouldn't make sense anymore: {
"extends": "./tsconfig.json",
"compilerOptions": {
"outDir": "./out-tsc/worker",
"lib": ["es2018", "webworker"],
"types": []
},
"include": [
"src/**/*.worker.ts",
"projects/fibonacci-webworker/src/lib/fibonacci.ts"
]
}The entry It seems there is still some kind of custom webpack magic required like described by worker-plugin... Can anyone of the Angular team tell us if this will change anytime soon? Or do you consider the execution environment of an algorithm to be a developer decision? So a library only provides the algorithms and interfaces, but a web-worker would execute the algorithms against given parameters? |
IMHO, this would be the preferred approach, by using web-workers in a library you are making that library platform dependent as it cannot run on Node.JS since web-workers are only available in the browser. I'd definitly suggest the decision on how to run/consume a method to the application developer. So far, I didn't see a completing enough use-case were web-workers in a library were the optimal solution. In some cases the best solution was not to use web-workers and shift the heavy data transformation and calculations to the backend. |
Server-side rendering could potentially be impossible, but when someone creates computation with web workers, they do fallback too.
A server that is paid by process time. Dumb servers. Offline availability. I don't see a reason why the decision should not be up to the developer. Forbidding the option in libraries does no sense. Edit: 62 people gave this issue "like" which means that at least 62 people were bothered enough by not being able to do it to vote on it. At least 62 people found a compelling reason why we would like to have this option. |
|
Could we get the vote for the feature request done? @anuglar-robot I feel like we would easily get 30 votes to pass this. |
alan-agius4
removed
the
freq1: low
angular-robot
bot
added
the
feature: under consideration
|
Have also a usecase where I need this. |
|
@muhamedkarajic meanwhile, I explained how to do it fairly easily here: #15059 (comment) |
|
I think your solution is great, but it would be ideal if library authors could encapsulate web workers within themselves while hiding that internal implementation from its consumers. |
|
Was just looking into the viability of web/shared workers In a data-processing-heavy enterprise solution. So those teams would not be able to build and configure workers of any kind in the wrapper "shell". It's 2023 almost and we are still stuck with single-thread apps in the era of 16 thread+ computers/devices. Perhaps libraries could expose assets/files - like when lazy loading modules with the router? I feel like a way for libraries to expose assets/js files to be accessible from the main app is needed - not just what is directly bundled under the app dist. |
|
I made a template to dynamically launch a library webworker. This project uses a reactjs application, but this may help. |
|
I am developping a 3D mapping library, I also need workers in libs.
And since when every single library in the world has to support NodeJS? In my case it's a complete non-sense, it has to run in a browser to load a WebGL canvas. The fact you didn't see use-cases doesn't mean there are none, only that you didn't search very hard 😉
Indeed all libs that want to manage their workers directly would benefit that. There are workarounds but it would be much better to have a native support. Anyway, thanks @maxime1992 for the suggested solution. There's another option described here: https://stackoverflow.com/questions/57072680/web-worker-in-angular-library that seems simpler and will try first. |
🚀 Feature request
Command (mark with an
x)Description
I would like to use Web Workers in an Angular 8 library project.In version 8.1.1 this seems to be supported for project type 'application' only:
Describe the solution you'd like
Generating a Web Worker should work for project type 'library' as well.Describe alternatives you've considered
Rewrite TypeScript worker code to JavaScript file.The text was updated successfully, but these errors were encountered: