[chore] Remove NativeValue and NativeType from the VM entirely #1361
Assignees
Milestone
Comments
|
I'll probably take on this sometime in the next few months, even just as an experiment (or as part of sogno), even just to see the reduction in LOC :) |
This was referenced Feb 26, 2024
thehowl
added a commit
that referenced
this issue
Mar 18, 2024
Split from #1695 for ease of reviewing. Related to #814. Merge order: 1. #1700 (this one!) 2. #1702 3. #1695 I scrapped the "linked identifier" feature of native bindings. The reasoning mostly stems from the changes subsequently implemented in #1695, as having "linked identifiers" means that: - the Gno linked types must have a different name from Go's if they are within the same package, so their names don't conflict after transpilation - in order for the "linked types" to match in the generated code, the AST has to be rewritten to make a type alias (ie. `type Address = crypto.Bech32Address`) - while still allowing the type to be "modifiable" by the std package, because we want to add our own methods -- this is not possible for imported types, obviously - and if we try removing methods, this [creates errors](https://drop.howl.moe/9ba0d53115-Screenshot_from_2024-02-27_23-50-34.png) because the methods on the original types don't match 1-1 those implemented in AST Although, I think a decent case can be made besides easing our (my) life in transpiling: - Under the hood, the linked type mechanism works with `Store.AddGno2GoMapping`. This uses gonative (read: the bane of my existence -- #1361). If we remove all linked types, we can still pass data between Go and Gno using type literals, which don't incur in naming conflicts. - This also makes the workings of "native bindings" clearer in general, as they don't have any special linked types (which were currently hardcoded in the misc/genstd source) ## Reviewing notes - `Banker` got severely refactored as it was mapping entire go interfaces into Go; it now uses simple elementary functions, with its old behaviour split between Go and Gno. - many other functions (std/native.gno) have also been changed so that their native function only uses primitive types (so everything that used an Address, now uses a string). - Due to the naming conflicts already mentioned, Go's Banker has been changed to `BankerInterface` so that it doesn't conflict. - AddGo2GnoMapping is unused in the codebase. This has been removed from Store to disencourage any further usage; removal of Go2Gno code is out of scope for this PR (see #1361)
7 tasks
This was referenced Oct 2, 2024
This was referenced Oct 17, 2024
ltzmaxwell
pushed a commit
that referenced
this issue
Nov 26, 2024
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).  - Fixes #1084 - Fixes #2826 (by addressing root cause of slowness) - Fixes #588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <[email protected]>
This was referenced Nov 28, 2024
6 tasks
r3v4s
pushed a commit
to gnoswap-labs/gno
that referenced
this issue
Dec 10, 2024
](https://lu.ma/open-source-saturday-torino){kind=link}
This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for `gno test`. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost". The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a [blog post on "diary of a gnome"](https://gno.howl.moe/faster-tests/), but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).  - Fixes gnolang#1084 - Fixes gnolang#2826 (by addressing root cause of slowness) - Fixes gnolang#588, only running `_long` filetests on master and generally speeding up test execution. ## Impact - Test context (tests and filetests) - Tests and filetests now share the same `test.Store` when running. Coupled with `test.LoadImports`, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.) - One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a `println` used in package initialization. - The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: `// SEND:`, and `std.TestIssueCoin`. Some test balances had to be updated. - Filetests - Running filetests in `gno test -v` now also prints their output. - Realm tests are no longer executed in `main.gno`, but in a filename following the name of the filetest; this changed some `// Realm:` directives. - The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty `//` in the tests. - `// Error:` directives are now treated the same as everything else; and are updated if the `-update-golden-tests` flag is passed. - Removed the `imports` metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly"). - Removed support for different "modes" of importing stdlibs; further removing support for gonative (gnolang#1361). The remaining gonative libraries are `os`, `fmt`, `encoding/json`, `internal/os_test` and `math/big`. This removes the `-with-native-fallback` flag from `gno test`. - Consequently, filetests ending with `_native.gno` have largely been removed, and those ending with `_stdlibs.go` have had their suffix removed. - Some files testing `gonative` types and functions which were only used in a small subset of these tests, have been removed. - Tests - `gno test`, for testing packages, created a `main_test.gno` file that is then called directly from the machine on each test. This creates two identifiers, `tests` and `runtest`, which could come into conflict with those defined by the tests themselves. We now call `testing.RunTest` directly, without an intermediary. - Exports in the normal tests (ie. defined in `pkg`) are now visible in the tests of `pkg_test`. This is most useful for some standard library tests, where there often is an `export_test.go` with the sole scope of exporting some internal functions and methods for testing in the "real" testing file. - `gno lint`, and other occasions where we use `issueFromError` (like when parsing errors in `gno test`), will now also print the column of the error if given. ## Summary of internal changes - pkg/gnolang - `TestFiles` is the new function running the filetests. - `eval_test.go` has been removed, moving some of its improvements over to `TestFiles`, and reducing the scope of the corresponding tests in `debugger.go`, to what it's actually supposed to test for. - As a consequence of removing all of type mappings in `imports.go`, I removed `SetStrictGo2GnoMapping` in favour of always being "strict", and not allowing defining native types of defined types any more. - The tests in `gonative_test` where primarily related to this, so I removed them. Similarly for `preprocess_test`. - `TestFunc` / `TestMemPackage` have been removed as redundant, including some of their features in `cmd/go/test.go` (like supporting exporting symbols in `_test.gno` files) - The `Store` no longer has `ClearCache` and `SetStrictGo2GnoMapping`; `ClearCache` now has a better substitute in `BeginTransaction`. - the `testing` stdlib no longer caches `matchPat` and `matchString` as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing. - The tests/ directory has been removed / moved to `gnovm/pkg/test`. This directory should eventually contain the internal code for `gno test` as well; but for now I wanted to clean `tests` to ensure it is a directory just for integration tests, rather than code and testing systems. - `TestMachine` and `TestStore` have been renamed to Machine and Store, to avoid redundancy with the `test` package name. - Removed plenty instructions in `gnovm/Makefile` as now most tests are just in `pkg/gnolang`. - I removed `MsgContext.Msg` as unused. It can be re-added later if necessary. - In the CI, tests are now run with `-short`, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: `bytes`, `strconv`, `regexp/syntax`. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything. --- [](https://lu.ma/open-source-saturday-torino) --------- Co-authored-by: Marc Vertes <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
This is nothing to work on yet, but rather we should wait until launch coding is almost complete, so that the codebase may settle a bit, such that the PR that removes NativeValue/NativeType can be referenced in the future for forks that want these things.
A prerequisite would be to get the code directory structure fixed, such as replacing gnovm/pkgs/gnolang with gnolang/pkgs/gnovm.
The text was updated successfully, but these errors were encountered: