-
Notifications
You must be signed in to change notification settings - Fork 987
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Errored subgraphs sometimes resume correctly after restart #3236
Comments
This is mysterious to me. Do you have the source code for the subgraphs that exhibited these issues? |
Not for Indeed I had the issue again on
|
It says "Critical error logged" but it doesn't actually log any CRIT messages? |
Ah I filtered only for ERRO. Here are the CRIT messages for
Between each of those entries, there is a graph-node restart and little bit more indexing happening. |
Seeing the same behaviour. Had to close an Enzyme allocation with last known good PoI as restarting the graph node was resulting in less and less time passing before a subgraph failure would occur. |
Thanks @aasseman and @cryptovestor21 - @otaviopace does any of the above ring a bell given changes in non-deterministic error handling recently? |
I'll try take a look this week 🙂 |
@aasseman I finally took a look at those. In I'm not sure how this can happen in an non-deterministic fashion. I'll try to run this version locally and see if I can index the same block multiple times until it happens. And for the Again, not sure how this can happen in a non-deterministic fashion. One thing that I'm curious about is, what RPC provider do you use? Also it might be helpful if I can have access to those IPFS files to deploy locally and debug. I can use the commits I've linked above, but I might be wrong in which version I'm using compared to what you're running. |
In case of For |
Do you want to request a feature or report a bug?
Bug
What is the current behavior?
Graph-node v0.25.0.
Some subgraph's indexing would hang up, declaring a deterministic error. But after one (or more) restarts of the graph-node, would resume with seemingly no issues. Observed on
QmTBxvMF6YnbT1eYeRx9XQpH4WvxTV53vdptCCZFiZSprg
(Enzyme) andQmf5XXWA8zhHbdvWqtPcR3jFkmb5FLR4MAefEYx8E3pHfr
(Graph network).If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
Index
QmTBxvMF6YnbT1eYeRx9XQpH4WvxTV53vdptCCZFiZSprg
orQmf5XXWA8zhHbdvWqtPcR3jFkmb5FLR4MAefEYx8E3pHfr
.What is the expected behavior?
The error should really be deterministic. Here it seems that due to unknown factors, a subgraph can be stuck, then unstuck.
Logs for
QmTBxvMF6YnbT1eYeRx9XQpH4WvxTV53vdptCCZFiZSprg
:Logs for
Qmf5XXWA8zhHbdvWqtPcR3jFkmb5FLR4MAefEYx8E3pHfr
:The text was updated successfully, but these errors were encountered: