wasm: re-use //export mechanism for exporting identifiers within wasm modules

[sbinet]

Right now, compiling the following piece of Go code:

package main

func main() {
	println("hello")
	println("1+3=", MyAdd(1, 3))
}

//export MyAdd
func MyAdd(i, j int32) int32 {
	return i + j
}

like so:

$> GOOS=js GOARCH=wasm go build -o foo.wasm ./main.go

will produce the following foo.wasm module:

$> wasm-objdump -h ./foo.wasm

foo.wasm:	file format wasm 0x1

Sections:

     Type start=0x0000000e end=0x00000048 (size=0x0000003a) count: 10
   Import start=0x0000004e end=0x000000ce (size=0x00000080) count: 6
 Function start=0x000000d4 end=0x0000042c (size=0x00000358) count: 854
    Table start=0x00000432 end=0x00000437 (size=0x00000005) count: 1
   Memory start=0x0000043d end=0x00000442 (size=0x00000005) count: 1
   Global start=0x00000448 end=0x0000047b (size=0x00000033) count: 10
   Export start=0x00000481 end=0x0000048f (size=0x0000000e) count: 2
     Elem start=0x00000495 end=0x00000acf (size=0x0000063a) count: 1
     Code start=0x00000ad5 end=0x000a3b7a (size=0x000a30a5) count: 854
     Data start=0x000a3b80 end=0x001290fd (size=0x0008557d) count: 7
   Custom start=0x00129103 end=0x0012f117 (size=0x00006014) "name"

especially:

$> wasm-objdump -j export -x ./foo.wasm

foo.wasm:	file format wasm 0x1

Section Details:

Export:
 - func[750] <_rt0_wasm_js> -> "run"
 - memory[0] -> "mem"

ie: the user can not control what is exported.
the recommended way to export something currently, is to use js.NewCallback.
this is - mainly - because one needs to setup a few things for the Go and js runtimes to cooperate nicely together.

I would argue that, in the same spirit than when one compiles the same main.go file with -buildmode=c-shared, it should be possible to achieve the same thing for GOOS=xyz GOARCH=wasm, and only export what is //export-ed.
initialization of the runtime(s) would be performed via the wasm module's start function.

(this is a re-hash of neelance/go#22.)

[nilslice]

I've hit the same rough patches after doing some "real" work with wasm in Go, and trying to integrate my module into a JS workflow.

To me, the primary issue is how Go functions are provided to JS. Presently, it seems that polluting the global object (window) with Go functions is the only route. The method I've found to keep collisions to a minimum is to declare a new map[string]interface{} as a "namespace" which translates to a JS Object. Then to use this object to add typed js.Funcs:

func main() {
	done := make(chan struct{})
	js.Global().Set("protolock", make(map[string]interface{}))
	module := js.Global().Get("protolock")
	module.Set("initialize",
		js.FuncOf(func(this js.Value, args []js.Value) interface{} {
			if args == nil {
				fmt.Println("initialize: not enough args")
				return nil
			}

			return toJson(initialize(args[0].String()))
		}),
	)
	module.Set("status", 
		js.FuncOf(func(this js.Value, args []js.Value) interface{} {
			if args == nil || len(args) < 2 {
				fmt.Println("status: not enough args")
				return nil
			}

			return toJson(status(args[0].String(), args[1].String()))
	}))
	<-done
}

While this works, it doesn't make for a very seamless integration with modern JS workflows. Ideally, we'd be able to generate at minimum a .d.ts file and stubs for the JS functions made available from the wasm module, to fit into the ES6 import style used by the majority of JS toolchains (webpack, rollup, etc).

wasm-pack does a pretty good job with this, but it's not clear to me how we can achieve a similar experience with Go. Ideally, there would be a common interface to populate the instance.exports object, and to generate those populated functions as stubs that wrap an invocation of instance.exports.myFunc in a JS package which can be imported by other JS code.

This would make the interoperability between wasm modules in a JS environment much simpler, hiding the fact that a function is run in wasm.

I'm not sure the //export mechanism is right, which would complicate go build, and lock Go into a particular way of doing things. I'm not opposed to using //export , but would like to consider an option that doesn't need to be as deeply integrated. Rather, some code-gen tooling to read Go AST, output JS stubs and definition files (in ES6 module format). The tooling would need to be aware of how the compiled Go code translates into accessible JS calls - and this is the part where I need to do more digging... I'm very interested in helping out here, but I think there is more discourse to be had around the workflow.

Do we have a working doc/wiki anywhere to track this? I couldn't find anything in an admittedly rather shallow search. Happy to kick it off if one does not exist yet.

[nilslice]

I'm not sure the //export mechanism is right, which would complicate go build, and lock Go into a particular way of doing things. I'm not opposed to using //export , but would like to consider an option that doesn't need to be as deeply integrated.

After more digging and a quick browse through @neelance's wasm implementation as well as the disassembled .wat of a Go compiled .wasm binary, it's much more clear that the //export semantics are definitely needed.. I have a hacked together PoC of some AST tooling that generates the JS bridge code, but its nowhere near the ideal. Having control of what is explicitly provided to the instance.exports is the most usable way for JS integration, and hopefully future wasm->wasm module imports.

[johanbrandhorst]

TinyGo exposes this functionality via a //go:export pragma. Personally I don't really like magic comments, especially when there is a way to do this in pure Go with syscall/js.

[justinclift]

//export vs //go:export seems like just a difference in the specific text fragment to use.

The meaning for each approach seems the same though: "This function should be included in the wasm exports list, and therefore callable from Javascript. Functions without it will not be".

@johanbrandhorst When you say there's a way to do it in pure Go with syscall/js, what are you meaning?

[aykevl]

//go:export is basically an alias for //export in TinyGo so you could use that as well.

[johanbrandhorst]

I mean that you can also do

js.Global().Set("myexport", js.FuncOf(func(this js.Value, args []js.Value) interface{} {
    //stuff
})

[sbinet]

the point of my original request is exactly to not only target and support js, but also browser-less environments, especially now with wasi picking up speed. (my own original use case being wagon)

[johanbrandhorst]

It'll be hard to say whether we'll have the same capability to set exports via syscall/wasi until work has begun on that, so we can't really judge whether we'll need the pragma for that use case either. Don't we all agree that magic comments are pretty smelly?

[dingchaoyan1983]

the wasm can't be used in webworker also.

[pkieltyka]

Why can’t the wasm be used in a webworker?

[dingchaoyan1983]

@pkieltyka because the exported function is mounted on window or global variable, but in webworker, we should add it on self

[Splizard]

Shouldn't standard Go exporting rules be used?
ie. all functions beginning with an uppercase letter would be exported.

[sbinet]

I was coming from the cgo machinery when I wrote that.
but you do have a point, even if there is a bit of an uncanny valley effect wrt the fact that, right now, a Wasm module needs (AFAIR) to be a main Go package.

taking a regular Go package (non-main), and populating the resulting Wasm module with the Go-exported functions using the regular Go-exporting rules would go a long way towards making the Go->Wasm experience more streamlined (and in line with the usual expectations of a Go developer).

[aykevl]

Note that exporting all Go symbols starting with an uppercase character may drastically increase file size, as the linker won't be able to do dead code elimination. I'm not sure whether this is happening right now, but it's something to keep in mind.

[slinkydeveloper]

the point of my original request is exactly to not only target and support js, but also browser-less environments, especially now with wasi picking up speed. (my own original use case being wagon)

Agree, i would love to experiment golang wasm target for extending a golang software through a plugin like mechanism, running it in wasm intepreters like https://github.com/go-interpreter/wagon, https://github.com/perlin-network/life or https://github.com/wasmerio/go-ext-wasm. Is this worth another issue?

[Splizard]

Note that exporting all Go symbols starting with an uppercase character may drastically increase file size, as the linker won't be able to do dead code elimination. I'm not sure whether this is happening right now, but it's something to keep in mind.

I would imagine only the Go symbols in the root/main package would need to be exported.

[JAicewizard]

Another possible option would be a compiler(or linker) argument with a list of functions that need to be exported.

[aykevl]

Please see #38248, which I believe solves the same issue in a different way.

[finnbear]

I have been adding exports to the global Go object, which is created by wasm_exec.js. Something does need to be done about this but ideally it would also generate bindings (like rust's wasm-bindgen) so you are not limited to exporting functions of specific numbers of int32, int64, f32, and f64.

[decibelcooper]

There are some really great thoughts on this issue, but I'm curious about the current momentum on this topic.

After scanning the open issues, I've come to the following two conclusions for myself:

1. There is a lot of sense in having a go build mode for the purpose of generating more idiomatic WASM modules. E.g., modules that export functions that are actually intended to be used in human-written code like JS in a browser. This has been mentioned by @sbinet here and @cherrymui in cmd/link: don't resolve WebAssembly imports at link-time #29108. Though there are other apparent pain points that I imagine could be addressed with such a build mode. Perhaps the use of a start function, or even factorization of the Go runtime into another module?
2. Go already has handy export semantics on the package level, and (as implied by @Splizard) these semantics can be used with a new build mode to determine what is exported in the WASM module. Presumably, only what is exported by the "root" package should be exported in the module. Perhaps there is still a need for special directives to give details, but there is also kind of a way to implicitly declare imports in a go build mode specific to idiomatic WASM modules: declare a function prototype without a body. AFAICT, this is being explored already in tinygo.

Summary: it is currently painful to turn Go code into useful and concise WASM. The lack of idiomatic use of modules appears to be at the heart of that pain.

[markdiener10]

Way late in the comment process, but the ability to mix host binary code along with WASM code linked into the same binary and not be forced to have 2 separate projects, each with its own GOOS= GOARCH= . Just my 2 cents here.

[jens1205]

I guess this ticket needs to be implemented, if we want to build a webassembly component in go, as the Imports and Exports of the webassembly module must be defined in the wasm file.

[stevegt]

@nilslice I know it has been a long, long time since you wrote this:

To me, the primary issue is how Go functions are provided to JS. Presently, it seems that polluting the global object (window) with Go functions is the only route.

I'm currently looking at how to make it possible to mix wasm modules compiled with different versions of Go or TinyGo in the same application. I can't see any way to do this using the stock wasm_exec.js as long as Go() itself is a global. I'm considering starting a Go proposal which would completely remove globalThis from wasm_exec.js. Do you recall if you tried that yourself way back when?

Extism completely bypasses this problem and enables both mixed-version as well as mixed-language apps via its plugins, right? Skimming through the code, it looks like you're using Rust as the runtime entrypoint, which gives you more flexibility in the first place versus what's implemented in Go's wasm_exec.js?

[nilslice]

@stevegt

I know it has been a long, long time since you wrote this

😄 wow, has it really been nearly 5 years!? ...a long time indeed!

To be honest, I'm not sure I tried TinyGo back then. I can see how what aiming to do is challenging though. If you're able to, it might help to drop support for anything syscall/js provides and only use wasip1 as the GOOS. In the browser, you'd polyfill WASI with something like https://github.com/bjorn3/browser_wasi_shim on the "host" side. Both TinyGo and Go can target this environment, but it's up to you to plumb through the imports to give the modules access to anything on the JS side.

Extism completely bypasses this problem and enables both mixed-version as well as mixed-language apps via its plugins, right? Skimming through the code, it looks like you're using Rust as the runtime entrypoint, which gives you more flexibility in the first place versus what's implemented in Go's wasm_exec.js?

Yes, Extism bypasses this probelm entirely enabling precisely what you mention. However, in the browser, we have a pure TypeScript implementation of the Extism ABI / runtime. So it doesn't require Rust at all. Extism was created to solve the problem you describe, and normalize the guest/host environment to provide consistency -- not dissimilar to how jQuery smoothed over the nuances between browser APIs back in the day.

We're currently in the midst of publishing a v1.0 of Extism and are near ready to release this JS SDK (which works universally in the browser, Node, Deno and Bun): https://github.com/extism/js-sdk

If you're interested, it would be a great time to try this out and share feedback. For a JS host, please use this branch if you're also using the latest version of our PDKs.

I don't mean to clog up this issue with Extism stuff, so if you'd like to chat more about it I'm always available on our Discord or on the repo issue tracker!

[johanbrandhorst]

I think this issue can be closed in favor of the new wasmexport proposal: #42372.

[cherrymui]

#65199 is implemented. Closing this as a dup. Thanks.

[Malix-Labs]

@cherrymui https://docs.github.com/en/issues/tracking-your-work-with-issues/marking-issues-or-pull-requests-as-a-duplicate#marking-duplicates 🙃