Add a polling block handler

[fubhy]

Currently, we have blockHandlers with and without filter. Additionally, it would be great to have a polling filter that runs every nth block (or time window).

E.g.

blockHandlers
  - handler: handleFundShutDown
    filter:
      kind: polling
      every: 24h

[milonite]

I'm following this. Would be a very useful feature for us as well.

[greenGreengo]

There is any plan for this? On L2 we track AMM pools, which result in bad performance (the number of events is thousands per block). For us, would be much simpler to call a view function every hour or so.

[jbrumwell]

👍 for us block handlers without filters are not an option and relying on external monitoring to perform these tasks, would be an great feature to have 👍

[azf20]

For a first implementation I think polling simply by block number makes the most sense:

blockHandlers
  - handler: handleFundShutDown
    filter:
      kind: block
      every: 100

I think this would require a new schemaVersion, but should not require a new apiVersion. This filter should be applicable across protocols (Ethereum, NEAR etc.).

Pseudologic

if( (block.number - dataSource.startBlock) % filter.every == 0 ) => runBlockHandler

[maoueh]

Just a quick note about the pseudo-code, NEAR and Solana can "skip" blocks, so this would not be "perfect" each 100 blocks if block are skipped at the trigger boundary (block #99 then block #101), worst case would be to never trigger (really unlikely)

[azf20]

Thanks @maoueh that is definitely a good edge case to keep in mind

[Data-Nexus]

@azf20 could we use the every: 100 as a way to adjust the size of blocks scanned then trigger at the end of the scan? That way if it only sees block 1 through block 99 it still triggers since this is the end of the range?

[azf20]

@Data-Nexus interesting that suggests it would run at every block at the chain head?

[Data-Nexus]

Hmm, that wouldn't be workable then since indexers reach the chainhead at varying points. Seems like it would have to be stateful and find where this.block is greater than the previous handledBlock by the amount stated in the every field. Thus if there was no block 100 it would next run on block 101.

[0xJem]

+1 to this!

Our use case:

We take treasury snapshots on Ethereum mainnet every epoch (8 hours) using an event handler.
We don't have the same setup on other chains (polygon, arbitrum, fantom) as we don't have a contract that fires an event every X hours. (It doesn't need to be 8.) I also haven't been able to find contracts on these chains that consistently fire an event every few hours. (e.g. chainlink has an event, but it's every few minutes!)

As a result, we have implemented a block handler that handles every 86,400th block: https://github.com/OlympusDAO/olympus-protocol-metrics-subgraph/blob/8a17a4c8bf5ccdafaf3ae9981e13527d6f3d56e4/subgraphs/arbitrum/src/treasury/Assets.ts#L20

This is... slow.

I imagine it would be faster to just configure the subgraph to call the handler every X minutes/hours or every X blocks.

[leoyvens]

@incrypto32 is picking this up! But now we need to decide on the exact semantics of every: N. A few possibilities, from simplest to most complicated:

1. Every block multiple of N.
2. Every N blocks, counting from the subgraph start block.
3. Every N blocks, counting from the data source start block.

We also need to decide if the trigger should always run on the data source creation block, as a special case.

[azf20]

Thanks @leoyvens and @incrypto32!

I think that 3. is the most intuitive. And I think it should run on the data source creation block - perhaps every: 0 only runs once on that first block - which would be an initialization handler #3221

[ubinatus]

@incrypto32 is picking this up! But now we need to decide on the exact semantics of every: N. A few possibilities, from simplest to most complicated:

1. Every block multiple of N.
2. Every N blocks, counting from the subgraph start block.
3. Every N blocks, counting from the data source start block.

We also need to decide if the trigger should always run on the data source creation block, as a special case.

Great @incrypto32 !

I also think 3 is the most intuitive. Also, would using a time window be out of scope? I think it could be more verbose for a user that have subgraphs deployed to multiple networks (blocks with different target time). Ofc, this could be handled by adding those block polling number to the networks.json config.

[jeffywu]

+1 Agree, option 3 is very intuitive and this would allow us to take historical snapshots via subgraphs. It's not possible on non-mainnet chains due to performance issues right now.

[Data-Nexus]

option 3 works for me as well!

We've recently run into a few limitations when utilizing a blockHandler on fast chains. The use of a blockHandler seems to limit the scan range of the indexer (going from scanning a range of 2000 blocks down to 100 if the block handler hasn't reached it's start block or 1 if the block handler is active).

While going through the implementation of this, can we also make a point to ensure that graph-node's block range selection isn't reduced below the every: x?

eg, the block range scanned would be reduced from 2,000 to 1800 here:

blockHandlers
  - handler: SixHourRefresh
    filter:
      kind: block
      every: 1,800

[milonite]

Agree option 3

+1 to this!

Our use case:

We take treasury snapshots on Ethereum mainnet every epoch (8 hours) using an event handler. We don't have the same setup on other chains (polygon, arbitrum, fantom) as we don't have a contract that fires an event every X hours. (It doesn't need to be 8.) I also haven't been able to find contracts on these chains that consistently fire an event every few hours. (e.g. chainlink has an event, but it's every few minutes!)

As a result, we have implemented a block handler that handles every 86,400th block: https://github.com/OlympusDAO/olympus-protocol-metrics-subgraph/blob/8a17a4c8bf5ccdafaf3ae9981e13527d6f3d56e4/subgraphs/arbitrum/src/treasury/Assets.ts#L20

This is... slow.

I imagine it would be faster to just configure the subgraph to call the handler every X minutes/hours or every X blocks.

for this you can use the chainlink event to keep track of the time (timestamp start), when it reaches approx 8hr, it triggers the snapshot

[0xJem]

for this you can use the chainlink event to keep track of the time (timestamp start), when it reaches approx 8hr, it triggers the snapshot

Is there a particular Chainlink event and/or feed that you have in mind? In my experience, the events are fired every few minutes, which is too often to make a significant difference in the indexing speed.