Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cp][aptos-release-v1.25] [1.23] Temporarily loosen backpressures, until module loading improvement speeds up blocks #15614

Open
wants to merge 1 commit into
base: aptos-release-v1.25
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Backport

This will backport the following commits from aptos-release-v1.23 to aptos-release-v1.25:

Questions ?

Please refer to the Backport tool documentation

Copy link

trunk-io bot commented Dec 16, 2024

⏱️ 1h 24m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
rust-unit-tests 25m 🟩
execution-performance / single-node-performance 24m 🟩
rust-move-tests 13m 🟩
execution-performance / test-target-determinator 5m 🟩
rust-doc-tests 5m 🟩
test-target-determinator 5m 🟩
rust-cargo-deny 2m 🟩
fetch-last-released-docker-image-tag 2m 🟩
check 1m 🟥
check-dynamic-deps 48s 🟩
semgrep/ci 32s 🟩
general-lints 25s 🟩
file_change_determinator 14s 🟩
file_change_determinator 11s 🟩
permission-check 3s 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
execution-performance / single-node-performance 24m 14m +67%

settingsfeedbackdocs ⋅ learn more about trunk.io

@bchocho bchocho enabled auto-merge (squash) December 17, 2024 19:31

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor Author

✅ Forge suite realistic_env_max_load success on c5dc17fd6c8cbf8d0430534c824c2984eced7466

two traffics test: inner traffic : committed: 14772.52 txn/s, latency: 2687.90 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3000 ms), latency samples: 5616820
two traffics test : committed: 99.95 txn/s, latency: 1392.36 ms, (p50: 1400 ms, p70: 1400, p90: 1500 ms, p99: 1600 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.551, avg: 1.494", "ConsensusProposalToOrdered: max: 0.329, avg: 0.293", "ConsensusOrderedToCommit: max: 0.377, avg: 0.358", "ConsensusProposalToCommit: max: 0.666, avg: 0.651"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.72s no progress at version 38492 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.60s no progress at version 2248018 (avg 0.60s) [limit 16].
Test Ok

Copy link
Contributor Author

❌ Forge suite framework_upgrade failure on e07212d19a17186e03260fcc1bce53b4ffbadc64 ==> c5dc17fd6c8cbf8d0430534c824c2984eced7466

Compatibility test results for e07212d19a17186e03260fcc1bce53b4ffbadc64 ==> c5dc17fd6c8cbf8d0430534c824c2984eced7466 (PR)
Upgrade the nodes to version: c5dc17fd6c8cbf8d0430534c824c2984eced7466
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 166.49 txn/s, submitted: 196.01 txn/s, failed submission: 0.44 txn/s, expired: 29.53 txn/s, latency: 2204.20 ms, (p50: 2100 ms, p70: 2700, p90: 3600 ms, p99: 4300 ms), latency samples: 52712
Test Failed: Waiting for nodes to catch up to target version and epoch (None, Some(9)) timed out after 20 seconds, current status: Ok([("validator-0", 53874, 8), ("validator-1", 53874, 8), ("validator-2", 53874, 8), ("validator-3", 53874, 8)])

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:463:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:396:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:289:92
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: aptos_forge::interface::swarm::SwarmExt::fork_check::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:200:14
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:93:63
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start
Trailing Log Lines:
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="bd124847-bf82-4119-8558-fb1b0766db89">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="framework_upgrade::framework-upgrade">
            <failure message="Waiting for nodes to catch up to target version and epoch (None, Some(9)) timed out after 20 seconds, current status: Ok([(&quot;validator-0&quot;, 53874, 8), (&quot;validator-1&quot;, 53874, 8), (&quot;validator-2&quot;, 53874, 8), (&quot;validator-3&quot;, 53874, 8)])

Stack backtrace:
   0: anyhow::error::&lt;impl anyhow::Error&gt;::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:463:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:396:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:289:92
   4: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: aptos_forge::interface::swarm::SwarmExt::fork_check::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:200:14
   6: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: &lt;aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:93:63
   8: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15614-1734464767-e07212d19a17186e032","timestamp":"2024-12-17T19:58:50.266749Z","message":"Deleting namespace forge-framework-upgrade-pr-15614: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15614-1734464767-e07212d19a17186e032","timestamp":"2024-12-17T19:58:50.266766Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-15614"}
Failed to run tests:
Tests Failed

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          10m
aptos-node-1-validator-0                1/1     Running     0          6m35s
aptos-node-2-validator-0                1/1     Running     0          12m
aptos-node-3-validator-0                1/1     Running     0          12m
forge-testnet-deployer-znq5h            0/1     Completed   0          12m
genesis-aptos-genesis-eforge228-x7m4r   0/1     Completed   0          12m

Copy link
Contributor Author

✅ Forge suite compat success on e07212d19a17186e03260fcc1bce53b4ffbadc64 ==> c5dc17fd6c8cbf8d0430534c824c2984eced7466

Compatibility test results for e07212d19a17186e03260fcc1bce53b4ffbadc64 ==> c5dc17fd6c8cbf8d0430534c824c2984eced7466 (PR)
1. Check liveness of validators at old version: e07212d19a17186e03260fcc1bce53b4ffbadc64
compatibility::simple-validator-upgrade::liveness-check : committed: 14974.61 txn/s, latency: 2270.91 ms, (p50: 2400 ms, p70: 2400, p90: 2500 ms, p99: 2700 ms), latency samples: 484960
2. Upgrading first Validator to new version: c5dc17fd6c8cbf8d0430534c824c2984eced7466
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7223.54 txn/s, latency: 3908.91 ms, (p50: 4300 ms, p70: 4500, p90: 4900 ms, p99: 5400 ms), latency samples: 131740
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7633.72 txn/s, latency: 4203.89 ms, (p50: 4500 ms, p70: 4600, p90: 5000 ms, p99: 5900 ms), latency samples: 254300
3. Upgrading rest of first batch to new version: c5dc17fd6c8cbf8d0430534c824c2984eced7466
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5927.30 txn/s, latency: 4887.12 ms, (p50: 5500 ms, p70: 5700, p90: 5800 ms, p99: 6000 ms), latency samples: 111780
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6027.75 txn/s, latency: 5524.32 ms, (p50: 6000 ms, p70: 6000, p90: 6200 ms, p99: 7100 ms), latency samples: 202920
4. upgrading second batch to new version: c5dc17fd6c8cbf8d0430534c824c2984eced7466
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11423.28 txn/s, latency: 2351.26 ms, (p50: 2600 ms, p70: 2700, p90: 2800 ms, p99: 2900 ms), latency samples: 209960
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 12284.34 txn/s, latency: 2593.42 ms, (p50: 2700 ms, p70: 2800, p90: 2900 ms, p99: 3400 ms), latency samples: 398820
5. check swarm health
Compatibility test for e07212d19a17186e03260fcc1bce53b4ffbadc64 ==> c5dc17fd6c8cbf8d0430534c824c2984eced7466 passed
Test Ok

Copy link

trunk-io bot commented Dec 17, 2024

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants