Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy subgraphs between shards #2313

Merged
merged 65 commits into from
Apr 21, 2021
Merged

Copy subgraphs between shards #2313

merged 65 commits into from
Apr 21, 2021

Conversation

lutter
Copy link
Collaborator

@lutter lutter commented Mar 25, 2021

With this PR, subgraphs can be copied from one shard to another shard. The system can now deal with multiple copies of the same deployment hash existing side by side. These copies need to be in different shards (there's little point in having the same data duplicated in the same database), and only one of these copies (the 'active' one) will be used to respond to queries.

Copying is initiated by running graphman copy create ..., and queries can be switched to the copy with graphman copy activate .. once the copy has finished. Behind the scenes, copying uses most of the grafting machinery from #2293. In addition, all operations make sure that they now uniquely identify the precise copy of a deployment using a DeploymentLocator on the write path for a subgraph (encapsulated in a WritableStore) For queries, the QueryStore automatically picks the active copy of a subgraph. Within the store, deployments are now identified by the id of the deployment_schemas table; DeploymentLocator encapsulates that so that the rest of the code can be oblivious to that.

The system is limited to 5 active copy/graft operations per index pod. In the future, it might be better to limit this systemwide. The limit is hardcoded, but it would be easy to make that configurable in graph-node.toml

While a copy/graft is under way, progress is printed approximately every 3-5 minutes; the progress message looks like

INFO Copied 68.36% of `UpdateOperator` entities (150000/219432 entity versions), 36.60% of overall data, 
         dst: sgd139, subgraph_id: QmXB2mf78FTeXPFT7R7LUiUGNxypTbDnoMXYGu7oRAfh9z, 
         component: SubgraphInstanceManager

A copy/graft can be cancelled by unassigning the destination deployment. That will (with some delay of up to 5 minutes) lead to the copy process stopping, and the subgraph being stopped.

A list of active copies can be generated with graphman copy list:

------------------------------------------------------------------------------
deployment           | QmQx9BPnarDU29c1vroiWCBd4wrZuMpngvA6peuSB2Kzy6
action               | sgd120 -> sgd141 (shard_1)
started              | 2021-04-06T00:17:41+00:00
progress             | 12.51% done, 310000/2477526
------------------------------------------------------------------------------
deployment           | QmRuorV4Ck1sVdpfpAAwfYXnf3cfSkbDwZvvzWud9SH8Dg
action               | sgd130 -> sgd145 (shard_1)
queued               | 2021-04-06T00:18:16+00:00
------------------------------------------------------------------------------
deployment           | QmRpG1kMLJxnruvCScfs4JYPeHJEnJk1ZsEdhpfqaGPUHs
action               | sgd19 -> sgd146 (shard_1)
cancel requested     | 2021-04-06T00:22:30+00:00
------------------------------------------------------------------------------
deployment           | QmSsv4sJNwyEkSmwWgErmHS4tneAuVEoChSRNcyXSE1mjG
action               | sgd43 -> sgd142 (shard_1)
cancelled            | 2021-04-06T00:22:56+00:00

The details of a copy operation can be printed with graphman copy status <dst> which prints something like

src          | 122
dst          | 144
target block | 12152039
duration     | 25m
status       | .

         entity type           |   next   |  target  |  batch   | duration
--------------------------------------------------------------------------
> Loan                         |   637909 |  1283075 |   144322 |      19m
. Poi$                         |        0 |   108543 |    10000 |      0ms
✓ Pool                         |   113975 |   113974 |   160000 |     112s
✓ Proxy                        |       22 |       21 |    20000 |     40ms

To help clarify the distinction between internal and external identifier of a deployment, I plan on renaming SubgraphDeploymentId to DeploymentHash. Since this is a very boring, but very intrusive change, I will do that in a separate PR; that's the reason why a lot of code now has variables hash: SubgraphDeploymentId.

This PR sits on top of #2293.

@lutter
Copy link
Collaborator Author

lutter commented Mar 31, 2021

I added a few commits so that connections that are used for copying, and therefore use fdw, come from a separate pool where connections are closed more aggressively. That achieves two things: (1) 'normal' connections don't have additional fdw conns hanging off them (2) the number of connections that are busy because of copying is limited separately so that heavy copy activity doesn't block other database work.

@lutter lutter force-pushed the lutter/cp branch 3 times, most recently from c951875 to b414c33 Compare April 2, 2021 20:47
@lutter lutter force-pushed the lutter/cp branch 5 times, most recently from e0a57a7 to 21f7c7e Compare April 6, 2021 23:04
@lutter lutter force-pushed the lutter/cp branch 2 times, most recently from 74cb23c to e8b9188 Compare April 7, 2021 23:05
lutter added 23 commits April 20, 2021 16:37
We now use one dedicated fdw connection, and hang on to it for the duration
of the copying.
The various assign commands and 'copy create' need to be able to find a
unique deployment. Allow passing a shard to disambiguate.
Also, mark the cancellation time in copy_state
The SubgraphStore gives access to all instances of a deployment, but for
the WritableStore we need to be very careful that we do not accidentally
query or modify another deployment instance than the one in the site. These
code changes will hopefully make it more obvious when we would rely on
SubgraphStore functionality that is dependent on the precise deployment
instance.
Starting a subgraph can be very slow if we have to copy the subgraph as
part of starting it. Also, call `Writable.start_subgraph_deployment` early
on so that dynamic data sources are in place when we look for them.
The id makes it easier to distinguish between log messages coming from
different copies of the same subgraph
@lutter lutter changed the base branch from lutter/graft to master April 21, 2021 20:30
@lutter lutter merged commit 01c5b8d into master Apr 21, 2021
@lutter lutter deleted the lutter/cp branch April 21, 2021 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants