Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Assertion about dynamic data sources failed #4994

Closed
1 of 3 tasks
paymog opened this issue Nov 16, 2023 · 14 comments · Fixed by #5083
Closed
1 of 3 tasks

[Bug] Assertion about dynamic data sources failed #4994

paymog opened this issue Nov 16, 2023 · 14 comments · Fixed by #5083
Labels
bug Something isn't working

Comments

@paymog
Copy link

paymog commented Nov 16, 2023

Bug report

A subgraph deployed into my graph node is failing with

thread 'QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC[2737]' panicked at 'assertion failed: self.hosts.last().and_then(|h| h.creation_block_number()) <=
    data_source.creation_block()', /graph-node/core/src/subgraph/context/instance.rs:136:9

I'm also seeing issues with the creation of runtime hosts.

Relevant log output

Nov 16 09:27:23.651 WARN no runtime host created, there is already a runtime host instantiated for this data source, address: 6261666b726569676875626d677572717070336b6d6f666c637663356635766c366661376b736f34657077737a366235366e73706d377669346c75, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
Nov 16 09:27:23.810 WARN no runtime host created, there is already a runtime host instantiated for this data source, address: 6261666b726569633671767073636c6d6f6570787473726e786d6f7a70326f3566777579756c6164753733773762677461366b35796c766c6c7779, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
Nov 16 09:27:31.856 WARN no runtime host created, there is already a runtime host instantiated for this data source, address: 6261666b726569676d32717264357366656c756a35367135783274673369706674676e6b7a6b37697a6b7a6e7868346a676f697965776d32697234, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
Nov 16 09:27:32.002 WARN no runtime host created, there is already a runtime host instantiated for this data source, address: 6261666b72656964643765377669706b36787978657078646a3468616268357a6e75717173326b65696e796b733433376f35326f69376536746561, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
thread 'QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC[2737]' panicked at 'assertion failed: self.hosts.last().and_then(|h| h.creation_block_number()) <=
    data_source.creation_block()', /graph-node/core/src/subgraph/context/instance.rs:136:9

IPFS hash

QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC

Subgraph name or link to explorer

No response

Some information to help us out

  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux
@paymog paymog added the bug Something isn't working label Nov 16, 2023
@paymog
Copy link
Author

paymog commented Nov 16, 2023

Some more logs. Here's what I see during indexer startup

zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:47.911 DEBG Subgraph started, start_ms: 14, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphRegistrar
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:47.920 INFO Resolve subgraph files using IPFS, n_templates: 9, n_data_sources: 7, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:47.922 INFO Resolve offchain mapping, link: /ipfs/QmdtYrMLRLGywMcJQZeSAyMjJ3ZfCVRaHj3v2AwFPtFuux, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:48.156 INFO Successfully resolved subgraph files using IPFS, features: nonFatalErrors, grafting, n_templates: 9, n_data_sources: 7, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:48.195 INFO Starting subgraph writer, queue_size: 5, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:49.101 INFO Data source count at start: 16951, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:49.138 INFO forcing subgraph to use static filters., sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:55.577 DEBG Requesting 0 block(s), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:55.581 DEBG Starting or restarting subgraph, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:55.582 DEBG subscribing to chain head updates, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:55.617 DEBG Starting block stream, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager

Here are 3 different instances of the panic happening

zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.721 DEBG Requesting logs for blocks [111450, 111949], contract 0x7777777f279eba3d3ad8f4e708545291a6fdba8b, 3 events, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.721 DEBG Requesting logs for blocks [111450, 111949], contract 0x7777777a456ff23d9b6851184472c08fbda73e32, 3 events, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.721 DEBG Requesting logs for blocks [111450, 111949], event 0xad59ebba8bfb06ba01a615a611467ca3bef86a275bd5e9704d3b295112550ba5, 2 contracts, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.722 DEBG Requesting logs for blocks [111450, 111949], event 0xa45800684f65ae010ceb4385eceaed88dec7f6a6bcbe11f7ffd8bd24dd2653f4, 2 contracts, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.722 DEBG Requesting logs for blocks [111450, 111949], contract 0x7777773606e7e46c8ba8b98c08f5cd218e31d340, 1 events, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.727 INFO Create data source, params: bafkreigu544g6wjvqcysurpzy5pcskbt45a5f33m6wgythpgb3rfqi3lzi, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.727 INFO Done processing trigger, gas_used: 5351268103, data_source: ZoraCreator1155Impl, thread 'QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC[2737]' panicked at 'assertion failed: self.hosts.last().and_then(|h| h.creation_block_number()) <=\n    data_source.creation_block()', /graph-node/core/src/subgraph/context/instance.rs:136:9
zora-indexer-2-graph-node-67889449d8-94mlc graph-node handler: handleSetupNewToken, total_ms: 10, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: SetupNewToken(indexed uint256,indexed address,string,uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.730 INFO Done processing trigger, gas_used: 270057097, data_source: ZoraCreator1155Impl, handler: handleUpdatedPermissions, total_ms: 1, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedPermissions(indexed uint256,indexed address,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.777 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.777 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.778 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.779 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.781 DEBG Found 0 relevant block(s), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-2-graph-node-67889449d8-94mlc graph-node Nov 16 18:38:00.785 DEBG Requesting 0 block(s), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream

and another

zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.408 DEBG Requesting logs for blocks [111450, 111949], event 0xa45800684f65ae010ceb4385eceaed88dec7f6a6bcbe11f7ffd8bd24dd2653f4, 2 contracts, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.408 DEBG Requesting logs for blocks [111450, 111949], contract 0x7777773606e7e46c8ba8b98c08f5cd218e31d340, 1 events, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: BlockStream
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.409 INFO Create data source, params: bafkreigu544g6wjvqcysurpzy5pcskbt45a5f33m6wgythpgb3rfqi3lzi, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.410 INFO Done processing trigger, gas_used: 5435812442, data_source: ZoraCreator1155Impl, handler: handleURI, total_ms: 6, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: URI(string,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.411 DEBG eth_call, block_number: 110450, block_hash: 63a1d30b01071f9abdf9d93ca6c41c667204480a237eb80e4f3535863cbcdb23, data: 0e89341c0000000000000000000000000000000000000000000000000000000000000001, address: 027232b4613e7e243e2ea57c37094226b9671f50, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.415 INFO Create data source, params: bafkreigu544g6wjvqcysurpzy5pcskbt45a5f33m6wgythpgb3rfqi3lzi, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.416 INFO Done processing trigger, gas_used: 5351268103, data_source: ZoraCreator1155Impl, handler: handleSetupNewToken, total_ms: 5, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: SetupNewToken(indexed uint256,indexed address,string,uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.417 INFO Done processing trigger, gas_used: 270057097, thread 'data_sourceQmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC[2737]: ' panicked at 'ZoraCreator1155Implassertion failed: self.hosts.last().and_then(|h| h.creation_block_number()) <=\n    data_source.creation_block(), ', handler/graph-node/core/src/subgraph/context/instance.rs: :handleUpdatedPermissions136:, 9total_ms
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node 1, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedPermissions(indexed uint256,indexed address,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.491 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.492 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.492 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.492 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-0-graph-node-78d7b9c565-tmrxg graph-node Nov 16 18:27:52.492 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager

and another

zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.630 INFO Create data source, params: bafybeicgolwqpozsc7iwgytavete56a2nnytzix2nb2rxefdvbtwwtnnoe/metadata, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.630 DEBG eth_call, block_number: 110450, block_hash: 63a1d30b01071f9abdf9d93ca6c41c667204480a237eb80e4f3535863cbcdb23, data: 13966db5, address: 027232b4613e7e243e2ea57c37094226b9671f50, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.634 DEBG eth_call, block_number: 110450, block_hash: 63a1d30b01071f9abdf9d93ca6c41c667204480a237eb80e4f3535863cbcdb23, data: a0a8e460, address: 027232b4613e7e243e2ea57c37094226b9671f50, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.639 DEBG eth_call, block_number: 110450, block_hash: 63a1d30b01071f9abdf9d93ca6c41c667204480a237eb80e4f3535863cbcdb23, data: e8a3d485, address: 027232b4613e7e243e2ea57c37094226b9671f50, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.648 INFO Create data source, params: bafybeicgolwqpozsc7iwgytavete56a2nnytzix2nb2rxefdvbtwwtnnoe/metadata, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.648 INFO Create data source, params: 0x027232b4613e7e243e2ea57c37094226b9671f50, name: ZoraCreator1155Impl, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.648 INFO Done processing trigger, gas_used: 15572115718, data_source: ZoraNFTCreatorFactory1155V1, handler: handleNewContractCreated, total_ms: 19, transaction: 0x2fb6…63ab, address: 0x6a35…6703, signature: SetupNewContract(indexed address,indexed address,indexed address,string,string,(uint32,uint32,address)), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.655 INFO Done processing trigger, gas_used: 251142831, data_source: ZoraCreator1155Impl, handler: handleUpdatedPermissions, total_ms: 6, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedPermissions(indexed uint256,indexed address,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.661 INFO Done processing trigger, gas_used: 489099942, data_source: ZoraCreator1155Impl, handler: handleUpdatedToken, total_ms: 6, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedToken(indexed address,indexed uint256,(string,uint256,uint256)), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.662 INFO Done processing trigger, gas_used: 176871672, data_source: ZoraCreator1155Impl, handler: handleUpdatedRoyalties, total_ms: 1, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedRoyalties(indexed uint256,indexed address,(uint32,uint32,address)), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.664 INFO Done processing trigger, gas_used: 259832522, data_source: ZoraCreator1155Impl, handler: handleOwnershipTransferred, total_ms: 1, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: OwnershipTransferred(address,address), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.668 INFO Done processing trigger, gas_used: 251142831, data_source: ZoraCreator1155Impl, handler: handleUpdatedPermissions, total_ms: 4, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedPermissions(indexed uint256,indexed address,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.672 INFO Done processing trigger, gas_used: 488981628, data_source: ZoraCreator1155Impl, handler: handleUpdatedToken, total_ms: 4, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedToken(indexed address,indexed uint256,(string,uint256,uint256)), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.676 INFO Done processing trigger, gas_used: 252614536, data_source: ZoraCreator1155Impl, handler: handleUpdatedPermissions, total_ms: 3, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedPermissions(indexed uint256,indexed address,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.678 DEBG eth_call, block_number: 110450, block_hash: 63a1d30b01071f9abdf9d93ca6c41c667204480a237eb80e4f3535863cbcdb23, data: 0e89341c0000000000000000000000000000000000000000000000000000000000000001, address: 027232b4613e7e243e2ea57c37094226b9671f50, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.682 INFO Create data source, params: bafkreigu544g6wjvqcysurpzy5pcskbt45a5f33m6wgythpgb3rfqi3lzi, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.683 INFO Done processing trigger, gas_used: 5435812442, data_source: ZoraCreator1155Impl, handler: handleURI, total_ms: 6, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: URI(string,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.684 DEBG eth_call, block_number: 110450, block_hash: 63a1d30b01071f9abdf9d93ca6c41c667204480a237eb80e4f3535863cbcdb23, data: 0e89341c0000000000000000000000000000000000000000000000000000000000000001, address: 027232b4613e7e243e2ea57c37094226b9671f50, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.688 INFO Create data source, params: bafkreigu544g6wjvqcysurpzy5pcskbt45a5f33m6wgythpgb3rfqi3lzi, name: MetadataInfo, sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.689 INFO Done processing trigger, gas_used: 5351268103, data_source: ZoraCreator1155Impl, handler: handleSetupNewToken, total_ms: 5, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: SetupNewToken(indexed uint256,indexed address,string,uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node Nov 16 18:39:59.690 INFO Done processing trigger, gas_used: 270057097, data_source: ZoraCreator1155Impl, handler: handleUpdatedPermissions, total_ms: 1, transaction: 0x2fb6…63ab, address: 0x0272…1f50, signature: UpdatedPermissions(indexed uint256,indexed address,indexed uint256), sgd: 2737, subgraph_id: QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC, component: SubgraphInstanceManager
zora-indexer-2-graph-node-5478c6c5c5-fwd9x graph-node thread 'QmbrEG3Ph195yb341q1utKWu1WiupaeSeYRGRTdZaDASqC[2737]' panicked at 'assertion failed: self.hosts.last().and_then(|h| h.creation_block_number()) <=\n    data_source.creation_block()', /graph-node/core/src/subgraph/context/instance.rs:136:9

@paymog
Copy link
Author

paymog commented Nov 27, 2023

This has now happened on a second subgraph completely unrelated to the one above in a different graph node cluster we run.

For the one above, we dropped it entirely (and removed all underlying data) then redeployed and everything worked perfectly. I suspect this is a regression in 0.33.0. @leoyvens any ideas here?

@paymog
Copy link
Author

paymog commented Nov 27, 2023

Actually this is happening for ~4 different subgraphs in our infra

image

@azf20
Copy link
Contributor

azf20 commented Nov 28, 2023

Thanks @paymog

For the one above, we dropped it entirely (and removed all underlying data) then redeployed and everything worked perfectly.

You are saying that this happens intermittently?

Is there any pattern to the subgraphs, i.e. is it affecting File Data Sources, or any dynamic data source?

@paymog
Copy link
Author

paymog commented Nov 28, 2023

yup, seems to happen intermittently and only on some subgraphs.

I went through the drop and redeploy flow mentioned above for another subgraph today and was surprised to be confronted with this error log after the redeployment

Failed to start subgraph, code: SubgraphStartFailure, error: internal constraint violated: deployment sgd31426 must have been created before calling initialize_block_ptr but we got NotFound, sgd: 32135, subgraph_id: QmXbNGPEF78kmKZWkrPDvVctiVQGshcLQCoXe2wk5B5sTa, component: SubgraphInstanceManager

Restarting the indexer fixed the issue. I've never seen this error log before though.

It seems to be happening with dynamic data sources and the assertion mentioned seems to be part of the dynamic data sources code path.

@leoyvens
Copy link
Collaborator

I know of two situations that can trigger this assert:

  1. Two Graph Nodes attempting to index the same deployment, in the same DB shard. This will cause race conditions and can manifest with that error.
  2. This bug we fixed a while ago fix(store): Fix rewinding of dynamic data sources #4810, the fix is included in the latest release so you shouldn't see this issue with subgraphs you deployed on the latest release.

@paymog
Copy link
Author

paymog commented Nov 28, 2023

I know of two situations that can trigger this assert:

  1. Two Graph Nodes attempting to index the same deployment, in the same DB shard. This will cause race conditions and can manifest with that error.
  2. This bug we fixed a while ago fix(store): Fix rewinding of dynamic data sources #4810, the fix is included in the latest release so you shouldn't see this issue with subgraphs you deployed on the latest release.

We're running v0.33.0 so I guess the PR should in our build.

How would it be that 2 graph nodes attempt to index the same deployment? In the original case and the one I just mentioned, there definitely are multiple nodes running in the cluster. I wonder if this might be related to #5011 and #5014 which are showing issues with assigning subgraphs to specific indexers.

@leoyvens
Copy link
Collaborator

Sometimes graphman commands are not very obvious, which graphman commands do you typically run to manage assignments, and what documentation do you read for them (so we can make it better)?

@paymog
Copy link
Author

paymog commented Nov 28, 2023

Sometimes graphman commands are not very obvious, which graphman commands do you typically run to manage assignments, and what documentation do you read for them (so we can make it better)?

We use the JSON API to remove and change assignments, specifically the subgraph_reassign and subgraph_remove APIs. I also use graphman reassign when trying to do manual pausing/reassigning (eg when debugging the issue in #5011). I know it would be better to use graphman pause but that doesn't seem to have an API we can build automation around.

That being said, we should probably move this specific thread to one of the other issues and keep this issue focused on the assertion failure.

@leoyvens
Copy link
Collaborator

subgraph_remove and subgraph_reassign should stop the subgraph without requiring a restart of the index node, but it's not immediate, it can take a few seconds. Restarting the node does guarantee that it will stop. There is one situation I'm not sure of though:

On subgraph_reassign, from index node A to node B, then A will be notified to stop the subgraph and B will be notified to start the subgraph. My question is, does B make sure A has stopped before starting? @lutter, do you know? If B doesn't wait, then a race condition could happen where both are indexing the deployment. Though I'd think the error would clear with a restart of both nodes.

@paymog
Copy link
Author

paymog commented Nov 28, 2023

subgraph_remove and subgraph_reassign should stop the subgraph without requiring a restart of the index node, but it's not immediate, it can take a few seconds. Restarting the node does guarantee that it will stop. There is one situation I'm not sure of though:

Yup, that was our experience as well on prior versions of graph node. We never encountered issues like this on v0.33.0. However, for some reason we don't always see that behavior now.

@paymog
Copy link
Author

paymog commented Dec 11, 2023

Happening on another subgraph, QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN which is a uniswap subgraph

goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:22.965 INFO Done processing trigger, gas_used: 20413920947, data_source: Factory, handler: handleNewPair, total_ms: 56, transaction: 0x7af8…b8d9, address: 0x5c69…aa6f, signature: PairCreated(indexed address,indexed address,address,uint256), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:22.965 INFO Done processing trigger, gas_used: 10397461, data_source: Pair, handler: handleTransfer, total_ms: 0, transaction: 0x7af8…b8d9, address: 0xeb19…1822, signature: Transfer(indexed address,indexed address,uint256), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.068 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: 70a08231000000000000000000000000891ad0385ce33d55f2dddd376421311cf8d2ca15, address: eb1961a9da9b793fd63a551961413a80b6b31822, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.081 INFO Done processing trigger, gas_used: 5674633517, data_source: Pair, handler: handleTransfer, total_ms: 115, transaction: 0x7af8…b8d9, address: 0xeb19…1822, signature: Transfer(indexed address,indexed address,uint256), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.082 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.087 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c6570000000000000000000000006b175474e89094c44da98b954eedeac495271d0f, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.093 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.104 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000dac17f958d2ee523a2206206994597c13d831ec7, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.115 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c6570000000000000000000000000000000000085d4780b73119b644ae5ecd22b376, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.121 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c6570000000000000000000000005d3a536e4d6dbd6114cc1ead35777bab948e3643, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.131 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c65700000000000000000000000039aa39c021dfbae8fac545936693ac917d5e7563, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.136 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c65700000000000000000000000086fadb80d8d2cff3c3680819e4da99c10232ba0f, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.142 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c65700000000000000000000000057ab1ec28d129707052df4df418d58a2d46d5f51, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.145 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c6570000000000000000000000009f8f72aa9304c8b593d555f12ef6589cc3a579a2, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.151 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000c00e94cb662c3520282e6f5717214004a7f26888, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.160 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000514910771af9ca656af840dff83e8264ecf986ca, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.166 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000960b236a07cf122663c4303350609a66a7b288c0, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.171 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000c011a73ee8576fb46f5e1c5751ca3b9fe0af2a6f, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.177 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c6570000000000000000000000000bc529c00c6401aef6d220be8c6ea1667f6ad93e, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.183 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000008ea60443d74180ae2f5aa5c850fc32717939c657000000000000000000000000df5e0e81dff6faf3a7e52ba697820c5e32d806a8, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.189 INFO Done processing trigger, gas_used: 80724582669, data_source: Pair, handler: handleSync, total_ms: 108, transaction: 0x7af8…b8d9, address: 0xeb19…1822, signature: Sync(uint112,uint112), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.253 INFO Done processing trigger, gas_used: 1146773346, data_source: Pair, handler: handleMint, total_ms: 63, transaction: 0x7af8…b8d9, address: 0xeb19…1822, signature: Mint(indexed address,uint256,uint256), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.255 DEBG eth_call, block_number: 11014158, block_hash: 35a15949ff371fcdd3cca88b2b441e0b005caa5061732c721162a8a06ef4da71, data: e6a439050000000000000000000000009b06d48e0529ecf05905ff52dd426ebec0ea3011000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2, address: 5c69bee701ef814a2b6a3edd4b1652cb9cc5aa6f, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.260 INFO Done processing trigger, gas_used: 5668712465, data_source: Pair, handler: handleSync, total_ms: 7, transaction: 0xbb3e…967a, address: 0x5fa7…03e3, signature: Sync(uint112,uint112), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node thread 'QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN[9645]Dec 11 12:56:23.286' panicked at 'assertion failed: self.hosts.last().and_then(|h| h.creation_block_number()) <=\n    data_source.creation_block() ', INFO/graph-node/core/src/subgraph/context/instance.rs :136Done processing trigger:9
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node , gas_used: 1527666181, data_source: Pair, handler: handleSwap, total_ms: 26, transaction: 0xbb3e…967a, address: 0x5fa7…03e3, signature: Swap(indexed address,uint256,uint256,uint256,uint256,indexed address), sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.321 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager
goldskyraw-indexer-5-graph-node-8488447cf6-8klms graph-node Dec 11 12:56:23.321 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 9645, subgraph_id: QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN, component: SubgraphInstanceManager

How can we fix this without deleting all of the underlying data? This is especially problematic for subgraphs that take a very long time to index, like uniswap.

@paymog
Copy link
Author

paymog commented Dec 11, 2023

@leoyvens is it safe to downgrade from 0.33.0 to 0.32.0?

@leoyvens leoyvens mentioned this issue Dec 16, 2023
Merged
@paymog
Copy link
Author

paymog commented Jan 16, 2024

@leoyvens will a new patch version with this fix be released anytime soon?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix data source rewind across more than one block graphprotocol/graph-node
3 participants
@paymog @azf20 @leoyvens