Medusa vs Echidna gas usage

[rappie]

Medusa's gas usage metrics seem to be higher than Echidna.

I'm testing this using:

        uint256 gasBefore = gasleft();
        doSomething();
        fl.log("gas used", gasBefore - gasleft());

Here's an example output for the same function call with a complex state:

• Echidna: 688957
• Medusa: 716057

There is also the possibility that somehow the state is built different due to small differences between medusa/echidna's setup or initial values. I'm trying to have everything exactly the same, like setting block number/timestamp to 1 before deploying anything.

Is this a known issue? Any tips for debugging?

[ggrieco-tob]

I'm not sure gas usage in hevm is very precise. I think medusa should give a more realistic estimation compared to geth.

[elopez]

Are you using any cheatcodes as part of doSomething? I'm not sure how they're priced in medusa, but that might be different.

[rappie]

I'm not using any cheatcodes. I'm trying to keep it as clean as possible.

It's a pretty straightforward function that just does a lot, based on a complex state. Things like looping over arrays, calculating shares, transfering tokens, etc.

[rappie]

I'm debating whether or not to do a more precise benchmark, and also including Foundry. Would this be helpful? Or is this information already available somewhere?

[rappie]

Here are the benchmark results:

• Echidna: 592767
• Medusa: 601767
• Foundry: 601817

Looks like Medusa is very accurate (assuming Foundry is accurate).

Repo:
https://github.com/rappie/fuzzer-gas-metric-benchmark

Any feedback is welcome :)