Feature/fix firehose dynamic data source #4075
Conversation
|
Does this reproduce with static filters on? In principle the static filter should retrieve all the triggers that could potentially apply to a template, regardless of the dynamic data sources that are actually instantiated. Edit: turns out we don't always want to have static filters on with firehose, and will do it adaptively as introduced in #4008 |
sduchesneau
force-pushed
the
feature/fix-firehose-dynamic-data-source
branch
2 times, most recently
from
c53a20d to
29c277d
Compare
Exactly, this problem happens only without static filters, but since the performance of static filters is very bad on some use cases, the adaptive approach is much more appropriate, so both ways must work without determinism issues |
sduchesneau
force-pushed
the
feature/fix-firehose-dynamic-data-source
branch
from
29c277d to
1667e12
Compare
* This firehose bump requires all firehose endpoints to be upgraded to v2 * the Dynamic DataSource fix requires the ethereum firehose endpoint to expose the new sf.firehose.v2.Fetch.Block method, used to refetch block before apply new filters * added BlockRefetcher interface for mocking in ethereum integration tests
sduchesneau
force-pushed
the
feature/fix-firehose-dynamic-data-source
branch
from
ecbf583 to
b622682
Compare
|
(note: current unit test failing happen on master) |
|
When we mix rpcs and firehoses in a config, it would be good to avoid calling the firehose endpoint for a refetch on a subgraph that's using an rpc blockstream. But we don't have to deal with that right now. |
This PR leverages the feature of firehose-ethereum to fetch a single block (https://github.com/streamingfast/proto/blob/develop/sf/firehose/v2/firehose.proto#L13-L40) at current cursor in its full version.
It includes a bump in the firehose protocol from
sf.firehose.v1.Stream/Blockstosf.firehose.v2.Stream/BlocksFirehose endpoints now have a minimal version requirement to because of the upgrade to sf.firehose.v2.Stream/Blocks:
43ffb0f(streamingfast/firehose-near@43ffb0f)When a dynamic data source is added on a block, that block must be fetched back from firehose before we apply the new source to it, because the block that we have in hands is incomplete (transformed by the firehose filter to only contain interesting transactions).