add a lambda transactional-test, clean up test driver a bit to improve coverage of existing tests #15463

brmataptos · 2024-12-03T18:47:59Z

Description

Add a move-compiler-v2/transactional-tests/tests/llambda/ directory and a config for running lambda tests there.
Changed transactional-tests/tests/tests.rs to

allow running with a specific compiler version
add a few comments to make it easier to figure out how to add a config
added a config baseline which is run for most things (not /operator_eval/ or /access_control/)
fixed a few tests which weren't being run with baseline before and should be

How Has This Been Tested?

Ran tests.

Key Areas to Review

Please ignore contents of lambda test output, I pulled this out of the lambda stack to get the config fixed faster.

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-12-03T18:48:04Z

⏱️ 47m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-move-tests	13m	🟩
rust-move-tests	12m	🟩
rust-cargo-deny	7m	🟩 🟩 🟩 🟩
check-dynamic-deps	5m	🟩 🟩 🟩 🟩
rust-move-tests	3m	🟥
rust-move-tests	2m	🟥
general-lints	2m	🟩 🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩
file_change_determinator	46s	🟩 🟩 🟩 🟩
permission-check	12s	🟩 🟩 🟩 🟩
permission-check	11s	🟩 🟩 🟩 🟩
check-branch-prefix	1s	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

rahxephon89 · 2024-12-04T00:53:18Z

third_party/move/move-compiler-v2/transactional-tests/tests/tests.rs

+        name: "baseline",
+        runner: |p| run(p, get_config_by_name("baseline")),
+        experiments: &[],
+        language_version: LanguageVersion::V2_2,


Whould we have a const like "LATEST_UNSTABLE_VERSION" for language and set it to V2_2 or it is just specific for v2.2?

I really want a version like LanguageVersion::LAMBDAS but that seems like overkill.

Oops, that one should not be V2_2 but should be stable. Hold on.

…aseline test config to ensure we are running baseline for all tests.

vineethk · 2024-12-04T12:44:46Z

third_party/move/move-model/src/metadata.rs

@@ -17,8 +17,8 @@ use std::{
 };

 const UNSTABLE_MARKER: &str = "-unstable";
-pub const LATEST_STABLE_LANGUAGE_VERSION: &str = "2.1";
-pub const LATEST_STABLE_COMPILER_VERSION: &str = "2.0";
+pub const LATEST_STABLE_LANGUAGE_VERSION: LanguageVersion = LanguageVersion::V2_1;


This code does not compile (try running cargo build from the aptos-core root directory or take a look at the failing CI).

oops, checked tests only, forgot this is used elsewhere. added new consts LATEST_STABLE_X_VALUE instead of remaing the old one to avoid conflicts with existing code.

vineethk · 2024-12-04T12:46:09Z

third_party/move/move-compiler-v2/transactional-tests/tests/tests.rs

@@ -29,37 +29,47 @@ struct TestConfig {
    experiments: &'static [(&'static str, bool)],
    /// Run the tests with language version 1 (if true),
    /// or with latest language version (if false).
-    is_lang_v1: bool,
+    language_version: LanguageVersion,


Doc comments should be updated.

vineethk · 2024-12-04T12:51:10Z

third_party/move/move-compiler-v2/transactional-tests/tests/tests.rs

@@ -109,6 +183,8 @@ const SEPARATE_BASELINE: &[&str] = &[
    "no-v1-comparison/assert_one.move",
    // Flaky redundant unused assignment error
    "no-v1-comparison/enum/enum_scoping.move",
+    "/lambda/",


Could we add a comment for these new entries like in the cases above? Otherwise, it seems like "flaky redundant unused assignment error" applies to these two entries.

vineethk · 2024-12-05T15:08:16Z

third_party/move/move-model/src/metadata.rs

-    pub fn latest_stable() -> Self {
-        CompilerVersion::from_str(LATEST_STABLE_COMPILER_VERSION).expect("valid version")
+    pub const fn latest_stable() -> Self {
+        LATEST_STABLE_COMPILER_VERSION_VALUE


Not sure why we need this change. Now, to upgrade the latest stable language version, we need to change both LATEST_STABLE_LANGUAGE_VERSION and LATEST_STABLE_LANGUAGE_VERSION_VALUE, and make sure to keep them in sync.

Instead, in the previous version, you would only need to change the LATEST_STABLE_LANGUAGE_VERSION, and if it is set to the wrong string, tests would fails.

@brmataptos Just to make sure you saw this comment. The PR is ready to go once this is addressed.

We need a const value to use with the config structs in tests.rs. While we could use a string, that's needlessly indirect. I've The old definition of latest_stable() cannot be declared as const, due to the non-const nature of trait functions like from_str() (apparently const traits are currently "unstable", but also there may be types for which from_str() wouldn't be constant). In this case, defining a custom to_str() for CompilerVersion and LanguageVersion allows us to derive LATEST_STABLE_COMPILER_VERSION from LATEST_STABLE_COMPILER_VERSION_VALUE and have a const value we can store in the config.

github-actions · 2024-12-18T18:14:26Z

✅ Forge suite `compat` success on `34025227eb5367c449bf9dae3cd226fe5c187904` ==> `e76b96f9bd50c7179214bd95f03301ec8c02f48e`

Compatibility test results for 34025227eb5367c449bf9dae3cd226fe5c187904 ==> e76b96f9bd50c7179214bd95f03301ec8c02f48e (PR)
1. Check liveness of validators at old version: 34025227eb5367c449bf9dae3cd226fe5c187904
compatibility::simple-validator-upgrade::liveness-check : committed: 3156.26 txn/s, submitted: 3454.99 txn/s, failed submission: 62.68 txn/s, expired: 298.73 txn/s, latency: 2544.53 ms, (p50: 1200 ms, p70: 1500, p90: 1700 ms, p99: 15400 ms), latency samples: 210479
2. Upgrading first Validator to new version: e76b96f9bd50c7179214bd95f03301ec8c02f48e
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 3910.90 txn/s, latency: 7291.28 ms, (p50: 7000 ms, p70: 10500, p90: 10900 ms, p99: 11100 ms), latency samples: 67300
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6036.30 txn/s, latency: 5433.01 ms, (p50: 5700 ms, p70: 6100, p90: 6700 ms, p99: 7000 ms), latency samples: 206260
3. Upgrading rest of first batch to new version: e76b96f9bd50c7179214bd95f03301ec8c02f48e
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7047.78 txn/s, latency: 4012.15 ms, (p50: 4400 ms, p70: 4700, p90: 4900 ms, p99: 5000 ms), latency samples: 129220
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7402.40 txn/s, latency: 4424.35 ms, (p50: 4800 ms, p70: 4900, p90: 5000 ms, p99: 5100 ms), latency samples: 248540
4. upgrading second batch to new version: e76b96f9bd50c7179214bd95f03301ec8c02f48e
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11566.74 txn/s, latency: 2391.12 ms, (p50: 2500 ms, p70: 2700, p90: 3000 ms, p99: 3100 ms), latency samples: 199240
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11935.09 txn/s, latency: 2653.75 ms, (p50: 2600 ms, p70: 3000, p90: 3300 ms, p99: 3500 ms), latency samples: 387320
5. check swarm health
Compatibility test for 34025227eb5367c449bf9dae3cd226fe5c187904 ==> e76b96f9bd50c7179214bd95f03301ec8c02f48e passed
Test Ok

github-actions · 2024-12-18T18:15:39Z

✅ Forge suite `realistic_env_max_load` success on `e76b96f9bd50c7179214bd95f03301ec8c02f48e`

two traffics test: inner traffic : committed: 14718.85 txn/s, latency: 2702.61 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5596460
two traffics test : committed: 99.98 txn/s, latency: 1373.68 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 2400 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.621, avg: 1.570", "ConsensusProposalToOrdered: max: 0.331, avg: 0.294", "ConsensusOrderedToCommit: max: 0.306, avg: 0.296", "ConsensusProposalToCommit: max: 0.596, avg: 0.590"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.58s no progress at version 36750 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.61s no progress at version 2341826 (avg 0.61s) [limit 16].
Test Ok

github-actions · 2024-12-18T18:16:11Z

✅ Forge suite `framework_upgrade` success on `34025227eb5367c449bf9dae3cd226fe5c187904` ==> `e76b96f9bd50c7179214bd95f03301ec8c02f48e`

Compatibility test results for 34025227eb5367c449bf9dae3cd226fe5c187904 ==> e76b96f9bd50c7179214bd95f03301ec8c02f48e (PR)
Upgrade the nodes to version: e76b96f9bd50c7179214bd95f03301ec8c02f48e
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1285.37 txn/s, submitted: 1314.51 txn/s, failed submission: 3.52 txn/s, expired: 29.14 txn/s, latency: 2953.38 ms, (p50: 2100 ms, p70: 2400, p90: 3600 ms, p99: 20300 ms), latency samples: 116929
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1478.44 txn/s, submitted: 1480.92 txn/s, failed submission: 2.48 txn/s, expired: 2.48 txn/s, latency: 2227.33 ms, (p50: 2100 ms, p70: 2400, p90: 3000 ms, p99: 4200 ms), latency samples: 119440
5. check swarm health
Compatibility test for 34025227eb5367c449bf9dae3cd226fe5c187904 ==> e76b96f9bd50c7179214bd95f03301ec8c02f48e passed
Upgrade the remaining nodes to version: e76b96f9bd50c7179214bd95f03301ec8c02f48e
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1390.36 txn/s, submitted: 1394.26 txn/s, failed submission: 3.90 txn/s, expired: 3.90 txn/s, latency: 2158.12 ms, (p50: 1800 ms, p70: 2100, p90: 3300 ms, p99: 4800 ms), latency samples: 121240
Test Ok

brmataptos changed the title ~~add a lambda transactional-test. while modifying test config, add a baseline test config to ensure we are running baseline for all tests.~~ add a lambda transactional-test, clean up test driver a bit to improve coverage of existing tests Dec 3, 2024

brmataptos requested review from rahxephon89, fEst1ck and vineethk December 3, 2024 18:49

rahxephon89 approved these changes Dec 4, 2024

View reviewed changes

brmataptos added 2 commits December 3, 2024 20:03

add a lambda transactional-test. while modifying test config, add a b…

fd34ef9

…aseline test config to ensure we are running baseline for all tests.

oops, had LanguageVersion::V2_2 for baseline config

8625359

brmataptos force-pushed the brm-add-lambda-tests branch from 605dff3 to 8625359 Compare December 4, 2024 04:12

vineethk reviewed Dec 4, 2024

View reviewed changes

address Vineeth's comments

e23b9d2

brmataptos requested a review from vineethk December 4, 2024 21:52

vineethk reviewed Dec 5, 2024

View reviewed changes

fix const declarations to only automatically stay in sync

dd7a1a6

brmataptos requested a review from vineethk December 10, 2024 16:27

vineethk approved these changes Dec 12, 2024

View reviewed changes

brmataptos enabled auto-merge (squash) December 13, 2024 03:23

This comment has been minimized.

Sign in to view

Merge branch 'main' into brm-add-lambda-tests

6c837d5

This comment has been minimized.

Sign in to view

fix test baselines after rebase

e76b96f

This comment has been minimized.

Sign in to view

brmataptos merged commit 9a6cc6f into main Dec 18, 2024
47 of 48 checks passed

brmataptos deleted the brm-add-lambda-tests branch December 18, 2024 18:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add a lambda transactional-test, clean up test driver a bit to improve coverage of existing tests #15463

add a lambda transactional-test, clean up test driver a bit to improve coverage of existing tests #15463

brmataptos commented Dec 3, 2024 •

edited

Loading

trunk-io bot commented Dec 3, 2024 •

edited

Loading

rahxephon89 Dec 4, 2024

brmataptos Dec 4, 2024

brmataptos Dec 4, 2024

brmataptos Dec 4, 2024

vineethk Dec 4, 2024

brmataptos Dec 4, 2024

vineethk Dec 4, 2024

brmataptos Dec 4, 2024

vineethk Dec 4, 2024

brmataptos Dec 4, 2024

vineethk Dec 5, 2024

vineethk Dec 6, 2024

brmataptos Dec 7, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

add a lambda transactional-test, clean up test driver a bit to improve coverage of existing tests #15463

add a lambda transactional-test, clean up test driver a bit to improve coverage of existing tests #15463

Conversation

brmataptos commented Dec 3, 2024 • edited Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Dec 18, 2024

✅ Forge suite compat success on 34025227eb5367c449bf9dae3cd226fe5c187904 ==> e76b96f9bd50c7179214bd95f03301ec8c02f48e

github-actions bot commented Dec 18, 2024

✅ Forge suite realistic_env_max_load success on e76b96f9bd50c7179214bd95f03301ec8c02f48e

github-actions bot commented Dec 18, 2024

✅ Forge suite framework_upgrade success on 34025227eb5367c449bf9dae3cd226fe5c187904 ==> e76b96f9bd50c7179214bd95f03301ec8c02f48e

brmataptos commented Dec 3, 2024 •

edited

Loading

trunk-io bot commented Dec 3, 2024 •

edited

Loading

✅ Forge suite `compat` success on `34025227eb5367c449bf9dae3cd226fe5c187904` ==> `e76b96f9bd50c7179214bd95f03301ec8c02f48e`

✅ Forge suite `realistic_env_max_load` success on `e76b96f9bd50c7179214bd95f03301ec8c02f48e`

✅ Forge suite `framework_upgrade` success on `34025227eb5367c449bf9dae3cd226fe5c187904` ==> `e76b96f9bd50c7179214bd95f03301ec8c02f48e`