[META] improved native bindings

[thehowl]

This issue has been hijacked from its original proposal to point to a "roadmap" of tasks to be done in the context of the below proposal, following the merge of the initial implementation in #859. The original, full proposal can be found below.

• Decide on a better approach to recovering previous StaticBlocks when precompiling fails: refactor(gnolang): handle duplicate method decls using TryDefineMethod #1459
• Support precompilation of native functions by linking directly to the Go source in stdlibs/, removing the necessity of stdshim. feat(transpiler): transpile gno standard libraries #1695
• Supporting the test-only tests/stdlibs/ directory in gno doc; as well as documenting native functions explicitly by adding the // injected comment after their declaration.
  Note: probably will be removed, see feat: std re-organization #2425
• Bring over the strconv package to a Gno implementation; this way in tests/imports.go the special packages and native values are restricted only to ImportModeNativePreferred, while everything else uses documentable native bindings also for tests. Additionally, the "package injector" system can officially be removed after this is done. feat(stdlibs): add package strconv #1464
• Document more standard library functions, especially in std! (feat: std re-organization #2425)
• Understand //gno:link as a "compiler directive" for types like std.Address, which are currently manually added as mappings in misc/genstd/mapping.go (no longer necessary)
• Create shims for native testing functions which are currently unsupported
• Improve performance/codegen of natively bound functions

Eventually, connected to this endeavour but a bit separate, we should shoot for removing all usages of NativeType/NativeValue: #1361

---

Proposal

This is an issue to propose and discuss a better integration and implementation of native Go bindings in the Gno standard libraries, trying to remove places for pitfalls and for erroneous behaviour while making it easier to contribute and expand functionality in the standard libraries.

Background

See also: #808, #668, #522

The standard library is seemingly designed to allow for running Gno code both in a context where stdlibs are loaded as .gno files from the gnovm/stdlibs directory, as native bindings (mostly useful for the filetests in gnovm/tests), and also where code is precompiled. This is a bit disorganised in different places, and has generated the issues mentioned above.

• stdlibWhitelist in gnovm/pkg/gnolang/precompile.go has to be kept in sync with the libraries from the Go stdlib which we have implemented. Although furthermore we also have to specify custom rules on how to handle the "std" package, and crucially would currently require more manual intervention if we wanted to precompile ie packages importing crypto/chacha20 (which is a custom Gno library).
• Many functions implemented in stdlibs/stdlibs.go are defined using pn.DefineNative when this is not really required because they don't intrinsically need to know about the context, but they do something which would be hard/slow to do in pure gno, with such an example being the sha256.Sum256 function.
• The test environment exposes more functions in a manner that is not really transparent - these are defined in gnovm/tests/imports.go, including packages like os, fmt, base64, etc. While it largely makes sense for these not to be exposed in standard gnovm contexts, it should be (1) be more visible to tooling (like gno doc) that these functions exist and that (2) they are available when writing tests.

Proposal

• I propose to remove native bindings the way they are done right now, and instead use a similar mechanism's to Go's with assembly code: if a function only has a declaration without a body, then it is expected to be implemented in Go.
  • Because of Gno's interpreted nature, this is only expected and supported in the standard library, where we can create tooling to do static code analysis and pregenerate all bindings to native Go functions.
  • Additionally, a new set of standard libraries is added to tests/stdlibs, which exposes the functions that are overridden or added in test contexts (gno test, gno run...).
• Declared Gno functions may be implemented in Go code by functions which have the same function signature (ie. arguments of the same type, but allowing for arguments with different name).
  • Unexported Gno functions may also have a corresponding native function, which should however have the name prefixed with X_ (ie. func println -> func X_println). This is to allow the function to be exported at a go level, and callable from another package.
• Additionally, another accepted signature is prefixing the function's arguments with the argument *gno.Machine (of pkg gnovm/pkg/gnolang). This allows to access gnovm internals in the same way that it is currently done in gnovm/stdlibs/stdlibs.go.
• If a non-stdlib type is used (ie. a named type or a complex type that references a named type), then the named types used should have a matching Go binding, through use of the //gno:link <gnoType> [<goPkg>.]<goType> compiler directive (see below for an example), in a similar fashion to how Go has //go:linkname.
• This has the advantage of maintaining a single source of truth for the standard libraries to find all the exported symbols available, and is transparent to documentation and AST-based tools like gno doc, and any possible future autocompletion tools where there won't be a need to add ad-hoc exceptions for native functions.
• The precompiler performs three kind of translations from a native function call to the precompiled Go version. See below for an example of all three.
  1. If the corresponding native function is of the kind func (*gno.Machine, ...), as described above, then the package is added as an import and the function is called with the first argument (*gno.Machine) set to nil.
  2. If the corresponding native function is a single-instruction return statement, the call to the native function is substituted with the value of that return statement.
  3. Otherwise, like in i., the package is added as an import and the function is called with the matching arguments.

Example std package

// gnovm/stdlibs/std/std.gno
package std

// Examples taken from real-world std functions

//gno:link Address "github.com/gnolang/gno/tm2/pkg/crypto".Bech32Address

type Address string

func EncodeBech32(prefix string, bytes [20]byte) (addr Address)

// Documentation which would show up in `gno doc`...
func GetOrigCaller() Address

func Hash(bz []byte) [20]byte

// gnovm/stdlibs/std/std.go
package std

import (
	gno "github.com/gnolang/gno/tm2/pkg/gnolang"
	"github.com/gnolang/gno/tm2/pkg/crypto"
	"github.com/gnolang/gno/tm2/pkg/bech32"
)

func EncodeBech32(prefix string, bytes [20]byte) crypto.Bech32Address {
	/* Note this check is done in stdlibs/stdlibs.go but would be useless here;
	before calling native functions like this one, where the type is [20]bytes,
	the vm/native calling system would do this check automatically.

	if len(bytes) != crypto.AddressSize {
		panic("should not happen")
	}*/
	b32, err := bech32.ConvertAndEncode(prefix, bytes)
	if err != nil {
		panic(err) // should not happen
	}
	return crypto.Bech32Address(b32)
}

func GetOrigCaller(m *gno.Machine) crypto.Bech32Address {
	if m == nil {
		panic("cannot call std.GetOrigCaller from precompiled code")
	}
	return m.Context.(ExecContext).OrigCaller
}

func Hash(bz []byte) [20]byte {
	return gno.HashBytes(bz)
}

Example user of std package, and generated precompiled code

// examples/gno.land/r/r1/r1.gno
package r1

import "std"

func SampleFunction() {
	std.EncodeBech32("r1", [20]byte{})
	std.Hash([]byte(std.GetOrigCaller())
}

// examples/gno.land/r/r1/r1.gno.gen.go (ie. precompiled)
package r1

import (
	gno "github.com/gnolang/gno/tm2/pkg/gnolang"
	"github.com/gnolang/gno/gnovm/stdlibs/std"
)

func SampleFunction() {
	std.EncodeBech32("r1", [20]byte{})
	// std.Hash is "inlineable" and thus auto-converted to gno.HashBytes.
	// std.GetOrigCaller takes in *gno.Machine, but we're not in the vm now so the machine is nil.
	gno.HashBytes([]byte(std.GetOrigCaller(nil))
}

[tbruyelle]

For someone who has often felt confused by the issues you mention, I really appreciate you took the time to investigate the whole topic and gave us a proposal of improvement!

I like the idea of using the function without body for that. So you can define the body whether in go directly or via the Init function. If you use Init it's because you want to access the context right ?
Maybe it would be nice to allow an alternate function signature that takes the gno.Machine as the first parameter, thus we can remove the injection via Init.
If I retake your example, this would also be valid mapping :

// json.gno
func MarshalJSON(v interface{}) ([]byte, error)

// json.go
import (
  "encoding/json"
  gno "github.com/gnolang/gno/gnovm/pkg/gnolang"
)

func MarshalJSON(m *gno.Machine, v interface{}) ([]byte, error) {
    return json.MarshalJSON(v)
}

But maybe that's a little far fetched and/or wouldn't work properly.

Otherwise, if the native functions are more complex, then the package "$gnoimportpath/gnovm/stdlibs/encoding/json" (for instance) is imported and the native function called.

I still don't really understand why do you mean by that. What the kind of complexity are you referring to?

[thehowl]

I like the idea of using the function without body for that. So you can define the body whether in go directly or via the Init function. If you use Init it's because you want to access the context right ? Maybe it would be nice to allow an alternate function signature that takes the gno.Machine as the first parameter, thus we can remove the injection via Init.

That idea also passed through my mind. I guess it could work, though I don't have the expertise to know if adding m *gno.Machine as the first argument covers most cases. From what I can see in stdlibs/stdlibs.go, pn is not used inside of the native functions themselves and store should also be available through m.Store where necessary, so I agree with you that it could be a viable solution, but I'd like to have some extra feedback here.

One concern is that precompilation (I explained better below how that works) would probably have more trouble if they're implemented this way. With the Init function, we can instead provide a Gno version and a precompiled version (like we're doing now for instance for std - where references to the std package are replaced with stdshim when precompiled)

Otherwise, if the native functions are more complex, then the package "$gnoimportpath/gnovm/stdlibs/encoding/json" (for instance) is imported and the native function called.

I still don't really understand why do you mean by that. What the kind of complexity are you referring to?

If the matching native function simply refers, for instance, to another Go standard library function, the easiest thing to do when precompiling Gno code is to replace the call to the function in the Gno stdlib to that in the Go stdlib

// stdlibs/strconv/strconv.gno

func Itoa(i int) string

// stdlibs/strconv/strconv.go

func Itoa(i int) string {
	return strconv.Itoa(i)
}

Then when we use strconv.Itoa in gno code, when we precompile it it is replaced with the go stdlib equivalent.

// example.gno
import "strconv"

func Render() string { return strconv.Itoa(11) }

// example.gno.gen.go
import "strconv"

func Render() string { return strconv.Itoa(11) }

However, say we change Itoa to also have other side effects, say a fmt.Println

// stdlbs/strconv/strconv.go

func Itoa(i int) string {
    fmt.Println("Hello!")
    return strconv.Itoa(i)
}

To maintain consistency between precompiled and gno execution, the precompiled version should become this one:

// example.gno.gen.go
import "github.com/gnolang/gno/gnovm/stdlibs/strconv"

func Render() string { return strconv.Itoa(11) }

[ajnavarro]

I think that maybe we don't need any precompile step to do, and we can just execute native code when interpreting the code on the VM if we detect that the function called is from the standard library. We will need to give to these functions a specific gas fee too.

[thehowl]

@ajnavarro we don't do precompile steps in normal executions, but precompilation is a feature supported by cmd/gno and I think we should continue to support.

And yes, indeed while we still need to sort out gas fees for native function it currently works like you said. Except that now everything's in a big file containing all the native functions (mostly stdlibs/stdlibs.go)

[tbruyelle]

One concern is that precompilation (I explained better below how that works) would probably have more trouble if they're implemented this way.

Yes probably, let's do it with function without body + Init first. Then we'll see if we can simplify with a only function without body.

Thx for the explanation.

[thehowl]

One thought @tbruyelle -- I just thought about it and your approach may actually be better and the issue I was referring to easily resolvable. We can simply make calls to the functions which use *gno.Machine use nil when precompiled, so ie. this is how I would implement AssertOriginCall

package std

import // ...

func AssertOriginCall(m *gno.Machine) {
  if m == nil {
    panic("std.AssertOriginCall called from precompiled code")
  }
  if len(m.Frames) != 2 {
    m.Panic("invalid non-origin call")
  }
}

Then when we precompile std.AssertOriginCall() in gno code, it gets converted to std.AssertOriginCall(nil).

[tbruyelle]

Indeed nice approach !
I never touched the precompile part, do you think it's easy to convert method calls like that ?

[thehowl]

I'm also not exactly knowledgeable about precompile internals, but starting from an AST tree and pregenerated mappings this should be doable without too much magic :)

[moul]

Related with #239 and #498

Good idea, but let's create a PoC to make informed decisions for next steps.

[thehowl]

I've incorporated the latest ideas in the comments into the OP, and also added a lengthier example section showing exactly how I envision both stdlib code and realm code (and precompiled).

Feedback is very welcome. I'll start working on a PoC soon.

[ajnavarro]

Leaving a different approach here, just to continue the discussion and checking pros and cons:

using the following example:

Example user of std package, and generated precompiled code

// examples/gno.land/r/r1/r1.gno
package r1

import "std"

func SampleFunction() {
	std.EncodeBech32("r1", [20]byte{})
	std.Hash([]byte(std.GetOrigCaller())
}

// examples/gno.land/r/r1/r1.gno.gen.go (ie. precompiled)
package r1

import (
	"github.com/gnolang/gno/gnovm/stdlibs/std"
)

func SampleFunction() {
	std.EncodeBech32("r1", [20]byte{})
	std.Hash([]byte(std.GetOrigCaller())
}

• std package is implemented in go, so we can use it as a dependency for go precompiled code, and internally on the VM.
• There will be no difference between the go and gno code, so the only thing that will change is the import.
• When code is executed inside the VM, we can use reflection to call functions on specific packages, std in this case.
• No need to keep track of a huge list of methods on the VM, any change to std package will be reflected on VM and generated go code.
• not crossed dependencies VM <-> std package

WDYT?

[thehowl]

@ajnavarro

One question, from your example: how does std.GetOrigCaller() gather access to *gno.Machine? Especially for the std package, many function calls require access to the VM's data. Otherwise, the proposals look very similar

[ajnavarro]

We could call something like std.With(vm).GetOrigCaller() from the VM to make it available. When the code is executed from Go, we can leave a default mocked VM as the default or use any other implementation. But these might be ideas for the future. std.GetOrigCaller() will call under the hood to std.With(&MockVM{}).GetOrigCaller() (or a nil VM) when called from Go, and that call to std.GetOrigCaller() will be translated to std.With(vm).GetOrigCaller() when called from Gno VM using reflection.

What I'm trying to do here is to avoid having code generation, and make everything as straightforward as possible to allow future IDEs to autocomplete and things like that.

Feel free to discard the idea if you feel it doesn't fit well (you have a bigger context here than me). I'm just hoping that this proposal helps to validate the main one.