ZKR_DB_KEY_NOT_FOUND while proving batch 75948 on Bali

[ToniRamirezM]

It looks like an issue with the witness generation. It was detected on CDK-Erigon v2.60.0-beta9.

https://app.datadoghq.com/logs?query=host%3Agke-zkevm-bali-pool-prover-8de10c2f-tgnj.europe-west1-d.c.prj-polygonlabs-zkevm-test.internal%20service%3Azkevm-bali%20container_id%3Abb3ef65d6725c7bde954e60429e7e1384fa290cddc5ec20e034910f770c2a0aa%20filename%3A1.log&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&context_event=AZMmpCZWAAAq-BxZD_qnuQAK&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Cdesc&to_event=AwAAAZMmpCNv1GOUtgAAABhBWk1tcENaV0FBQXEtQnhaRF9xbnVRQVIAAAAkMDE5MzI2YTctNzU1Zi00NmMzLTkxNTQtZDU4OWY0YjI2ODE2AAAUaw&viz=&from_ts=1731519811468&to_ts=1731520111472&live=false

[hexoscott]

Some notes from the investigation so far:
The problem is with batch 75948 - looking at the batchL2Data for this batch there is no L1 info tree index change. This is important because the missing slot is related to writing the L1 Block Hash to the GER manager contract. This is only written to if we use a new index at the sequencer.

The prover logs show this as the missing information from the witness:

A=0:0:0:a40d5f56:745a118d:906a34e:69aec8c0:db1cb8fa B=0:0:0:0:0:0:0:3 C=17f78b8d:30f15ffe:e2432660:db3ab8af:ebdac70d:db400edd:b329e010:4952e40e D=0:0:0:0:0:0:0:20 E=0:0:0:0:0:0:0:58

I ran a binary search for when this slot was changed on Bali and it was updated in block 6958864. This block belongs to batch 74981, so 966 batches prior to the one the prover is having trouble with which seems strange that it would need this slot at all.

My hunch at the moment is that this issue is somehow related to the info tree index changes. The problem on the prover stems from the changeL2Block routine which seems to confirm this theory.

[Sharonbc01]

Agreed to close this issue on 11/19.
Francesc can't reproduce the prover issue. execution worked in prover mode.
Toni also ran it again and it worked.

[ToniRamirezM]

Reopening the issue as it happened again on Bali for batch 79374. First batch on forkID 12, but this happened previously on forkID 11, so it looks like it is not related.

Prover logs and witness can be found here: https://gist.github.com/ToniRamirezM/32c38a587cbd2eae9a740caab8d9611b

[Sharonbc01]

Moving this from current milestone as not required for milestone.

[Sharonbc01]

Hi @fractasy did you gain nay more insight on this issue re thread here https://0xpolygon.slack.com/archives/C05SVKVLZ1A/p1733940581915119

cc: @ToniRamirezM