bug: flaky gas snapshot for some fuzz tests

[cruzdanilo]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (467aff3 2024-05-17T00:17:24.004804000Z)

What command(s) is the bug in?

forge snapshot

Operating System

macOS (Apple Silicon)

Describe the bug

for some fuzz tests (not sure which ones), forge fails to generate consistent gas snapshots, even when using the --force flag. this prevents it from being relied upon for ci validation (forge snapshot --check). the same behavior happens on macos (apple silicon) and linux (x86-64).

i created a somewhat minimal reproduction repository here: https://github.com/cruzdanilo/foundry-snapshot-repro

this is one output generated by for i in {1..16}; do forge snapshot --force | grep runs; done

[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)

this was run on github ci, here is the report.

[grandizzy]

I run the test and can see 2 different values for fuzzed UserOperation input, e.g. in 2 different runs showing

[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 417326, ~: 417135)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 417308, ~: 417135)

there are 2 different fuzzed cased values, run 164

UserOperation {
    0x00000000000000000000000000000000000000000000000000000000000022e9,
    0x0000000000000000000000000000000000000000000000000000000000002733,
    9764,
    12673873082346130924691454452779514193164883897088292420374917853190248779330,
    10006
}

vs

UserOperation {
    0x00000000000000000000000000000000000000000000000000000000000022e9,
    0x0000000000000000000000000000000000000000000000000000000000002733,
    9764,
    18542991761951108740563055453066386026290576689311603472268584080832751656013,
    10006
}

and run 244

UserOperation {
    0x0000000000000000000000000000000000000000000000000000000000002733,
    0x0000000000000000000000000000000000000000000000000000000000000e69,
    1638867016,
    801,
    12673873082346130924691454452779514193164883897088292420374917853190248779331
}

vs

UserOperation {
    0x0000000000000000000000000000000000000000000000000000000000002733,
    0x0000000000000000000000000000000000000000000000000000000000000e69,
    1638867016,
    801,
    18542991761951108740563055453066386026290576689311603472268584080832751656014
}

which explains why the difference

Similar issue is reported in
#5199
#5689
#6179

[grandizzy]

@cruzdanilo this happens due to fuzzing from mutated state values which are not deterministic, if adding locally

    /// forge-config: default.fuzz.dictionary-weight = 0
    function test_snapshot(UserOperation calldata userOp) external {

results are always [PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407510, ~: 407539)

Can you pls try and check if consistent results?