Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebWorker support? #312

Open
ebrensi opened this issue Aug 3, 2020 · 54 comments
Open

WebWorker support? #312

ebrensi opened this issue Aug 3, 2020 · 54 comments

Comments

@ebrensi
Copy link

ebrensi commented Aug 3, 2020

Hi I just discovered esbuild, coming from Parcel. One thing I like about parcel is that if I instaniate a WebWorker with the string literal filename, like

const ww = new Worker ('./myWorker.js')
ww.postMessage('work!')

Parcel will recognize the Worker constructor and create another bundle starting at ./myWorker.js. It also handles cache busting for the filename. So the bundle would be called ./myWorker8a68r8912q.js or something and that string would be updated in the code above.

Does ESBuild do something like that? If not, where would I look to implement that?

@evanw
Copy link
Owner

evanw commented Aug 3, 2020

Oh, interesting. That's pretty cool. I didn't know about that feature of Parcel. I assume this also works for the SharedWorker constructor. You should be able to get this to work with the current version of esbuild by just adding myWorker.js as an additional entry point in the list of entry points passed to esbuild.

However, the cache busting part won't work yet. None of the entry points for esbuild are cache-busted right now. I'm working on enabling this as part of general code splitting support (see #16) but it's still a work in progress. Introducing cache busting in entry point names is also a breaking change and I've been waiting to do this until the next batch of breaking changes. Once that's in, I think it should be relatively straightforward to make something like this work.

@ggoodman
Copy link

ggoodman commented Aug 9, 2020

This sort of feature is sorely needed in Rollup and can be achieved in Webpack with something like worker-plugin.

Having this built into esbuild as an opt-in would be absolutely fantastic! I think deferring to plugins would likely not do it 'right' as these would need to parse an AST, figure out bindings, etc.. (or just RegEx and yolo).

@ggoodman
Copy link

ggoodman commented Aug 9, 2020

There are other web apis that would benefit from this form of non require, non import()-based code splitting like audio worklets.

@rtsao
Copy link
Contributor

rtsao commented Sep 25, 2020

I think this would be a really nice feature.

As a stopgap, I was thinking this could be implemented as an esbuild plugin. This would impose some rather unfortunate syntax, but in a pinch, something like this would work:

import workerUrl from "workerUrl(./worker.js)";

const worker = new Worker(workerUrl)
import path from "path";
import { build } from "esbuild";

let workerLoader = (plugin) => {
  plugin.setName("worker-loader");
  plugin.addResolver({ filter: /^workerUrl\((.+)\)/ }, (args) => {
    return { path: args.path, namespace: "workerUrl" };
  });
  plugin.addLoader(
    { filter: /^workerUrl\((.+)\)/, namespace: "workerUrl" },
    async (args) => {
      let match = /^workerUrl\((.+)\)/.exec(args.path),
        workerPath = match[1];
      let outfile = path.join("dist", path.basename(workerPath));
      try {
        // bundle worker entry in a sub-process
        await build({
          entryPoints: [workerPath],
          outfile,
          minify: true,
          bundle: true,
        });

        // return the bundled path
        return { contents: `export default ${JSON.stringify(workerPath)};` };
      } catch (e) {
        // ...
      }
    }
  );
};

@blixt
Copy link

blixt commented Oct 12, 2020

Besides path resolution within new Worker(…) etc, it would also be nice to enable an entrypoint to be generated as a worker script, which means it will not use import * as x from "./x.js" but rather importScripts("./x.js") – at least until all browsers implement new Worker(…, { type: "module" }) (whatwg/html#550).

@gkjohnson
Copy link

gkjohnson commented Dec 29, 2020

As of version 5.x webpack can bundle web workers out of the box with the following syntax:

new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

import.meta.url is needed so the worker file is explicitly loaded relative to the current file. Otherwise it's loaded relative to the href of the page. And of course type: module is needed in order to optionally support modules. IMO it's the right way to support bundling Workers that because it promotes practices that work in browsers, as well. Parcel doesn't support this yet but I'm still hoping they will in v2.

This bundler looks great and I'd really look forward to seeing this feature added! It's been too hard to use WebWorkers generically for too long...

@endreymarcell
Copy link

endreymarcell commented Mar 5, 2021

Update: I created a minimal demonstration for how this plugin would work at https://github.com/endreymarcell/esbuild-plugin-webworker

An important difference to Webpack is that this one does not inline the code of the web worker script into the main bundle. Instead, it is built as a separate JS file and has to be shipped next to the main bundle.

@RReverser
Copy link

An important difference to Webpack is that this one does not inline the code of the web worker script into the main bundle. Instead, it is built as a separate JS file and has to be shipped next to the main bundle.

Webpack does not inline it either, it's emitted as a separate file too.

@endreymarcell
Copy link

@RReverser oh OK, thanks for the correction. I updated the original comment to not mislead anyone.
Also created a little example at https://github.com/endreymarcell/esbuild-plugin-webworker for anyone who might find it useful.

@dy
Copy link

dy commented Mar 14, 2021

Side note: it's possible to (partially) minify worker code with the inline-worker technique:

let worker = new Worker(URL.createObjectURL(new Blob([
  (function(){

  // ...worker code goes here

  }).toString().slice(11,-1) ], { type: "text/javascript" })
));

so as a workaround, workers can be stored in usual js files and included as follows:

// worker.js
export default URL.createObjectURL(new Blob([(function(){
  // ...worker code
}).toString().slice(11,-1)], {type:'text/javascript'}))
// main.js
import workerUrl from './worker.js'
let worker = new Worker(workerUrl)

@evanw
Copy link
Owner

evanw commented Mar 17, 2021

It works but uses a rather ugly hack to pass both the importer path and the imported file's path to be onLoad handler which makes me think there must be a better way...

The pluginData feature might make this easier.

Side note: it's possible to (partially) minify worker code with the inline-worker technique:

This is fragile and can easily break with esbuild. For example, setting the language target to es6 converts async functions into calls to a helper called __async that implements the required state machine using a generator function. However, that function is defined in the top-level scope and won't be available in the worker. So that code will crash.

As of version 5.x webpack can bundle web workers out of the box with the following syntax:

new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

This is my current plan to support this feature. This approach is sufficiently general and doesn't need to know anything about web workers. That would make this issue a duplicate of #795. It also looks like Parcel might drop support for the syntax that was originally proposed in this thread: parcel-bundler/parcel#5430 (comment). So that's another count against the original approach.

@RReverser
Copy link

RReverser commented Mar 17, 2021

That would make this issue a duplicate of #795.

Note that while there is some overlap, those are not really duplicates, because general asset handling and Worker handling need to differ.

In particular, with general asset handling Worker JS file would be treated as raw binary, and wouldn't get minified, the imports wouldn't resolve and so on - which is not what user normally wants. For this reason other bundlers have separate implementation paths for these two features.

@evanw
Copy link
Owner

evanw commented Mar 17, 2021

I was assuming that the new URL('./some/file.js', import.meta.url) syntax should cause a new JavaScript entry point to be created independent of whether there is new Worker nearby or not. The .js file extension is normally associated with the js loader so it wouldn't be interpreted as binary.

@RReverser
Copy link

The .js file extension is normally associated with the js loader so it wouldn't be interpreted as binary.

Fair enough. That's not how it's handled in any of those bundlers AFAIK, but seems to be a sensible route too (probably also needs to include mjs, ts etc. that can result in JavaScript code as well).

@blixt
Copy link

blixt commented Mar 17, 2021

new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

This is my current plan to support this feature.

This would also support the version without { type: "module" } right? So the bundle would use importScripts(…) instead of ES Module import syntax?

While I would love for this to all be ES Modules, neither Firefox nor Safari have any support for ES Module workers, which would make esbuild unsuitable for Worker bundling in most cases if it only emits ES Module-based Worker scripts.

Firefox issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1247687
WebKit issue: https://bugs.webkit.org/show_bug.cgi?id=164860 (but it turns out it was very recently "FIXED" so it might hit technical preview soon! 🎉)

@RReverser
Copy link

While I would love for this to all be ES Modules, neither Firefox nor Safari have any support for ES Module workers, which would make esbuild unsuitable for Worker bundling in most cases if it only emits ES Module-based Worker scripts.

Not necessarily, the point of type: "module" is to tell bundler that it contains ESM, so that it can bundle Worker code as a separate entry point. It's not meant to preserve imports in the final bundle anyway.

@blixt
Copy link

blixt commented Mar 17, 2021

Not necessarily, the point of type: "module" is to tell bundler that it contains ESM, so that it can bundle Worker code as a separate entry point. It's not meant to preserve imports in the final bundle anyway.

I guess that makes sense, assuming it always bundles down into "old school" Workers. I'd expect the flag to be kept in the output though, as the choice can have pretty big effects on other things, for example <link rel="modulepreload"> tags in the HTML will only work for entrypoints bundled for an ES Module worker, so if the bundler gets to choose freely between the two outputs it wouldn't be possible to set up an HTML file with the right preloading logic, for example.

Basically, an author should be able to always pick "old school" workers for cross-browser support, but also optionally use ES Module workers for Chrome (which has supported this for a long time).

@RReverser
Copy link

I'd expect the flag to be kept in the output though, as the choice can have pretty big effects on other things

Usually it has to be removed precisely because it has effect on other things.

E.g. if bundler supports code splitting, then chunk loading usually has to be transformed down to importScripts, which is only available in regular Workers, and not when type:"module" is used. For this reason other bundlers remove this property.

@endreymarcell
Copy link

It works but uses a rather ugly hack to pass both the importer path and the imported file's path to be onLoad handler which makes me think there must be a better way...

The pluginData feature might make this easier.

Oh right. Thanks! I updated my implementation at https://github.com/endreymarcell/esbuild-plugin-webworker to use pluginData instead of the hack.

@NullVoxPopuli
Copy link

NullVoxPopuli commented May 13, 2021

for me, ESM works as a work build target for the most part when combined with 'bundle' except, I need to remove all export keywords.
Is there a way to do that?


I submitted a PR to microsoft/vscode#123739
so if that's resolved, I don't need a way to strip exports from workers 🙃

@pierre10101
Copy link

I managed to resolve this after following the docs at link.

@lgarron
Copy link
Contributor

lgarron commented Jun 20, 2021

```js
new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

This is my current plan to support this feature. This approach is sufficiently general and doesn't need to know anything about web workers. That would make this issue a duplicate of #795. It also looks like Parcel might drop support for the syntax that was originally proposed in this thread: parcel-bundler/parcel#5430 (comment). So that's another count against the original approach.

It seems Firefox and Safari are newly making progress on module workers. Given the hours I've wasted on CJS workarounds, I'd love for esbuild to be ready for this ahead of their releases so we can live in an ESM-only world. And for what it's worth, such behaviour would already be very useful right now, since it already works in node and one major browser (Chrome) — enough to use for local development and try out things ahead of other browsers being ready. 😃

But I more importantly for me, I would like to note that that it would be most useful if this was not specifically tied to the implicit/explicit global Worker constructor. For those of us writing libraries that also work in node, workarounds like like web-worker may be necessary. In that case, call sites may look something like:

new CustomWorkerConstructor( new URL( "./worker", import.meta.url ), { type: "module" } )

(It's possible to assign CustomWorkerConstructor to a variable named Worker in the current scope, so that the AST locally resembles a direct browser worker construction. But that's not always desirable.)

Handling this at the new URL() level instead of the new Worker() level would avoid the issue.

@RReverser
Copy link

Another update to this issue is that Emscripten on C++ side and wasm-bindgen-rayon on Rust side both now emit new Worker(new URL('...', import.meta.url)) pattern for Workers internally used for WebAssembly threads, so supporting this syntax would automatically make bundling Wasm easier, too.

@diachedelic
Copy link

It appears that Chrome has very recently implemented the sync import.meta.resolve(specifier) function, which is pretty much identical to new URL(specifier, import.meta.url).href. import.meta.resolve seems to be a more permanent solution. Here is a good writeup explaining it: https://gist.github.com/domenic/f2a0a9cb62d499bcc4d12aebd1c255ab.

lgarron added a commit to cubing/twsearch that referenced this issue Nov 30, 2022
…/debugging the dev page.

Uses the workaround from evanw/esbuild#312 (comment) , since evanw/esbuild#2508 has not landed.
lgarron added a commit to cubing/twsearch that referenced this issue Nov 30, 2022
…/debugging the dev page.

Uses the workaround from evanw/esbuild#312 (comment) , since evanw/esbuild#2508 has not landed.
@remcohaszing
Copy link
Contributor

It would be nice if the esbuild uses the worker exports condition in package.json by default.

@fabiospampinato
Copy link

fabiospampinato commented Jan 26, 2023

IMO it'd be cool if something like the following got implemented, which as far as I know is pretty much the cleanest web worker abstraction possible:

import {foo, bar} from './foo?worker';

await foo ( 123 );

Where every exported function is an async function. Now the type checker doesn't need to be aware that those functions will be executed in web worker, your code calling those functions doesn't need to know that either (unless in some cases if you are passing on transferrable objects), and neither the functions themselves really need to be changed, other than being marked as async.

Something like this can be implemented as a plugin, but something more tightly integrated with the bundler would work better.

@peterpeterparker
Copy link

I tried the various approach of this thread - thanks for all the great ideas - but the only solution that worked out for me once I embedded the lib that contains the worker within my apps was loaded the worker as base64 as displayed in the gist https://gist.github.com/manzt/689e4937f5ae998c56af72efc9217ef0 of @manzt

lgarron added a commit to cubing/cubing.js that referenced this issue Jun 6, 2023
Release notes:

- Switch purely to module workers.
  - As of [Firefox 114](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/114), all modern JavaScript browsers and runtimes with worker support are now able to instantiate module workers. This is important for performance, as it significantly improves worker loading for `cubing.js` (requiring ⅓ as much code as before): #214
  - Due to the heavy complexity and maintenance burden of alternatives to module workers, `cubing.js` is dropping support for "classic" workers and there is no longer a "catch-all" fallback when worker instantiation fails. In removing this fallback, have carefully made sure `cubing.js` scrambles will continue work out of the box with all modern browsers, as well as the main runtimes that support web worker (`node` and `deno`). We have also tested compatibility against major bundlers. However, if you are passing `cubing.js` through your own choice of bundler, then it must correctly bundle the worker "entry file" using one of the following:
    - [`import.meta.resolve(…)`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve) — this is the only "officially supported" way that we plan to support indefinitely, and the only one that will work without showing a warning in the JavaScript console.
    - One of two variants of using `new URL(…, import.meta.url)` as an alternative to `import.meta.resolve(…)`.
    - Using this workaround for `esbuild`: evanw/esbuild#312 (comment)
lgarron added a commit to cubing/cubing.js that referenced this issue Jun 6, 2023
Release notes:

- Switch purely to module workers.
  - As of [Firefox 114](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/114), all modern JavaScript browsers and runtimes with worker support are now able to instantiate module workers. This is important for performance, as it significantly improves worker loading for `cubing.js` (requiring ⅓ as much code as before): #214
  - Due to the heavy complexity and maintenance burden of alternatives to module workers, `cubing.js` is dropping support for "classic" workers and there is no longer a "catch-all" fallback when worker instantiation fails. In removing this fallback, have carefully made sure `cubing.js` scrambles will continue work out of the box with all modern browsers, as well as the main runtimes that support web worker (`node` and `deno`). We have also tested compatibility against major bundlers. However, if you are passing `cubing.js` through your own choice of bundler, then it must correctly bundle the worker "entry file" using one of the following:
    - [`import.meta.resolve(…)`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve) — this is the only "officially supported" way that we plan to support indefinitely, and the only one that will work without showing a warning in the JavaScript console.
    - One of two variants of using `new URL(…, import.meta.url)` as an alternative to `import.meta.resolve(…)`.
    - Using this workaround for `esbuild`: evanw/esbuild#312 (comment)
@AliMD
Copy link

AliMD commented Jun 14, 2023

@evanw
I love ESBuild and I'm waiting for you to officially support ESM capability in worker.
I really don't like having to migrate from ESBuild to another tools like Parcel.
Could you please give a timeline for this feature?
Thank you in advance.

@firien
Copy link

firien commented Jul 19, 2023

I do not advocate this approach, but if you are desperate, new-url-v0.18.13 , has new URL("./path/to/script.js", import.meta.url) support. This is based on work with @lgarron from #795. These new URLs are treated as dynamic imports so you need splitting option - which is still marked as WIP - You should have some good browser tests in your project if you want to use this.

I have included a devcontainer in the branch to make it simple to get going. If you don't know how to get a container going in vscode, um, read tutorial. Fire up the container and vscode should prompt you to install any additional go stuff you need. The follow assumes you are building for linux-x64 platform - change if need be.

make clean
make platform-linux-x64

new binary will be at ./npm/@esbuild/linux-x64/bin/esbuild. With ESBUILD_BINARY_PATH you can use this new binary with the esbuild from npm. The versions of npm esbuild and this new binary HAVE TO match, new-url-v0.18.13 is currently pinned to v0.18.13 so your package.json line needs to be "esbuild": "0.18.13"

If you copy ./npm/@esbuild/linux-x64/bin/esbuild out of container, and into your project at bin/esbuild - just add ESBUILD_BINARY_PATH=bin/esbuild to your current esbuild script and good luck!



note that I use on a project with only js. typescript is untested.

@lgarron
Copy link
Contributor

lgarron commented Jul 20, 2023

I do not advocate this approach, but if you are desperate, new-url-v0.18.13 , has new URL("./path/to/script.js", import.meta.url) support. This is based on work with @lgarron from #795. These new URLs are treated as dynamic imports so you need splitting option - which is still marked as WIP - You should have some good browser tests in your project if you want to use this.

Glad to see that the code from last year can still be adapted! 🤓
(Even if I can't use it. 😭)

It looks like you're trying to support bare package names in firien@196a279
This is an issue for new URL(…, import.meta.url) (because bare names are semantically identical to relative file paths for the URL constructor). But it's a great use case for import.meta.resolve(…), which is now supported in all major browsers and node (experimental)/deno/bun: #2866

lgarron added a commit to cubing/cubing.js that referenced this issue Jul 24, 2023
For example: `bun run src/bin/scramble.ts -- 333`

Thanks to oven-sh/bun#3645 and
oven-sh/bun#3669, we can use `bun` directly on
our source code (except the WASM parts). This requires a few changes:

- Move around the source code to account for the fact that `esbuild`
  does not have understand relative `new URL(…, import.meta.url)` or
  `import.meta.resolve(…)` references yet:
  evanw/esbuild#312 (comment)
  - This has the unfortunate side effect that some files have to move to
    the source code root. This isn't *bad* per se, but it breaks some
    assumptions while still relying on some other assumptions. I hope we
    can move the code back some time soon.
- Avoid using the trampoline workaround when we seem to be in a browser environment.
- Avoid assuming that the output of `await import.meta.resolve(…)` can
  be passed to `new URL(…)` (`bun` returns a bath without the `file:`
  protocol).
lgarron added a commit to cubing/cubing.js that referenced this issue Aug 9, 2023
We can't quite do this for the `esm` build, due to evanw/esbuild#312
lgarron added a commit to cubing/cdn.cubing.net that referenced this issue Sep 8, 2023
I would prefer to avoid hardcoding any paths except those that are published as CDN entry points, but `esbuild` still doesn't support `new URL(…, import.meta.url)` (evanw/esbuild#312) or `import.meta.resolve(…)` (evanw/esbuild#2866).
lgarron added a commit to cubing/cdn.cubing.net that referenced this issue Sep 8, 2023
…irefox.

Users of the CDN can't prevent the worker failure messages, so they are just noise (and a potential slowdown). This hardcodes the entry file path where the instantiator (which reliably ends up in a chunk for us) expects it: https://github.com/cubing/cubing.js/blob/4afc6d727549aadc1e82abf52d531565953e7a9a/src/cubing/search/worker-workarounds/index.ts#L11

I would prefer to avoid hardcoding any paths except those that are published as CDN entry points, but `esbuild` still doesn't support `new URL(…, import.meta.url)` (evanw/esbuild#312) or `import.meta.resolve(…)` (evanw/esbuild#2866).
@AliMD
Copy link

AliMD commented Nov 22, 2023

any news on this?

@noseratio
Copy link

noseratio commented Dec 9, 2023

I'm late to this thread, having faced this issue with an audio worklet. I eventually resorted to the following (rather primitive) workaround:

In esbuild settings:

    entryPoints: [
      './src/www/components/audioRecorder/index.mts',
      './src/www/components/audioRecorder/audioRecorderWorklet.mts'
    ]

In the audio worklet audioRecorderWorklet.mts:

export function getWorkletUrl() {
  return import.meta.url;
}

if (globalThis.AudioWorkletProcessor) {  
  const AudioRecorderWorklet = class extends AudioWorkletProcessor {
    // ... audio worklet class
  }
  
  registerProcessor('audio-recorder-worklet', AudioRecorderWorklet);
}

In the main file index.mts:

import { getWorkletUrl } from './audioRecorderWorklet.mjs'

//...

await audioContext.audioWorklet.addModule(getWorkletUrl());

This works, but the esbuild-generated audioRecorderWorklet.mjs looks like this:

import {
  getWorkletUrl
} from "../../chunk-GRFBZUB4.mjs";
import "../../chunk-4OBMG7SZ.mjs";
export {
  getWorkletUrl
};

And the URL returned by getWorkletUrl() actually points to ../../chunk-GRFBZUB4.mjs, while audioRecorderWorklet.mjs essentially becomes a redundant stub.

I hope someone (including my future self) may find this useful, but I'm still looking for a better way of doing this.

PS. Huge thank to Evan for creating and maintaining ESBuild. It's an immensely useful tool, which keeps on delivering for cases where similar tools often fall short.

@fasiha
Copy link

fasiha commented Aug 1, 2024

I saw this issue was still open and thought esbuild didn't support the new Worker approach with webworkers but #2439 makes me think esbuild does support this?

@firien
Copy link

firien commented Aug 2, 2024

@fasiha, I believe #2439 is for allowing esbuild to preserve the comment, so it can make it to webpack. (https://github.com/privatenumber/esbuild-loader)

@andrevenancio
Copy link

Any updates?

@mcharytoniuk
Copy link

mcharytoniuk commented Oct 10, 2024

This issue can also be worked around with explicitly passing some kind of build id to esbuild, for example:

# can be replaced with anything, build id, random hash, commit id, date
BUILD_ID = $(date +"%s%N")

esbuild \
    --asset-names="./[name]_$(BUILD_ID)" \
    --entry-names="./[name]_$(BUILD_ID)" \
    --bundle \
    --define:__BUILD_ID=\"$(BUILD_ID)\" \
    --define:__STATIC_FILES_PATH="http://mycdn" \ # replace with your path to static files
    controller_mycontroller.ts \
    worker_myworker.ts 

It will produce a worker file in the output directory named exactly like worker_myworker_$(BUILD_ID). Then other files
will know the exact filename of the worker, they can include it with:

new Worker(`${__STATIC_FILES_PATH}/worker_myworker_${__BUILD_ID}.js`)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests