Code

Copy subgraphs between shards#2313

Code

Merged

lutter merged 65 commits intomastergraphprotocol/graph-node:masterfrom lutter/cpgraphprotocol/graph-node:lutter/cpCopy head branch name to clipboardApr 21, 2021

+3,646-2,106

Commits

Commits on Apr 20, 2021

• store: Make deployment_schemas.id the primary key for assignments

  Show description for 69cfa4d
  lutter

  committed
  69cfa4d

• store: Use deployment_schemas.id as the primary key for subgraph_manifest

  lutter

  committed
  6aa59bf

• store: Make it impossible for non-store code to create a Site

  lutter

  committed
  e86be3e

• server: Don't pass a schema into WS connection, deployment id is enough

  lutter

  committed
  124fd01

• graph, server: Replace use of SubgraphStore.deployment_state_for_name

  lutter

  committed
  6150465

• node, server: Use a QueryStoreManager GraphQLSubscriptionServer

  Show description for 95e2886
  lutter

  committed
  95e2886

• graph, graphql, store: Eliminate SubgraphStore.deployment_state_from_XXX

  lutter

  committed
  cbf83dc

• graph, store: Remove SubgraphStore.get_proof_of_indexing

  lutter

  committed
  a4d8cf2

• graph: Remove SubgraphStore.find_one

  lutter

  committed
  dc9fc0b

• graph: Remove SubgraphStore.find

  Show description for 0b3ff4a
  lutter

  committed
  0b3ff4a

• graph: Remove SubgraphStore.network

  lutter

  committed
  eb472e5

• all: Introduce a WritableStore

  Show description for c63f812
  lutter

  committed
  c63f812

• all: Pass deployment id into SubgraphStore.writable()

  lutter

  committed
  11e1c1f

• all: Return a Result from SubgraphStore.writable

  lutter

  committed
  65e9aad

• all: Remove SubgraphDeploymentId from WritableStore methods

  lutter

  committed
  bba26e2

• graph, store: Remove StatusStore.supports_proof_of_indexing

  lutter

  committed
  35f39d4

• graph: Remove store argument from EntityCache.as_modifications

  lutter

  committed
  4f88afe

• graph, store: Add a SubgraphStore.least_block_ptr

  Show description for 4f3c7e8
  lutter

  committed
  4f3c7e8

• store: Check for JSONB subgraphs in Schema::try_from

  lutter

  committed
  48b03c0

• store: Introduce a marker type for deployment primary keys

  lutter

  committed
  8c0c0a3

• store: Make deployment_schemas.id primary key for unused_deployments

  lutter

  committed
  9fb8432

• store: Use DeploymentId internally to identify deployments

  lutter

  committed
  b0912fb

• store: Add an 'active' flag to deployment_schemas

  Show description for 143c229
  lutter

  committed
  143c229

• store: Remove unsued primary::find_existing_site

  lutter

  committed
  179e36f

• store: Make all lookups of site by deployment hash use the active site

  lutter

  committed
  c449471

• all: Give the network indexer special access for getting a WritableStore

  Show description for 1a52afc
  lutter

  committed
  1a52afc

• all: Identify deployments by both hash and internal id

  lutter

  committed
  6d2bf25

• all: Remove subgraph hash from QueryStore.is_deployment_synced()

  lutter

  committed
  8365b60

• all: Remove subgraph hash from QueryStore.block_ptr()

  lutter

  committed
  b9834e4

• all: Remove subgraph hash from QueryStore.hash_non_fatal_errors()

  lutter

  committed
  4594e60

• all: Remove ChainHeadUpdateListener from ChainStore

  Show description for 1eb8a1b
  lutter

  committed
  1eb8a1b

• node: Clean up graphman setup and StoreBuilder

  lutter

  committed
  5609869

• all: Add a command 'graphman copy create'

  lutter

  committed
  873da41

• graph, server: Move explorer::Cache to graph::util::TimedCache

  lutter

  committed
  519b9a5

• graph: Parametrize TimedCache on the type of the key

  lutter

  committed
  ef56557

• store: Cache sites for a limited amount of time

  lutter

  committed
  7b48eb5

• node, store: Add graphman copy activate

  lutter

  committed
  009536f

• store: Change deployment_schemas to allow same deployment in different shards

  lutter

  committed
  fc7dab1

• store: Add a simple test for cross-shard copying

  lutter

  committed
  7e46552

• Document the test setup for single and sharded testing

  lutter

  committed
  e218b5c

• store: Cancel an ongoing copy when a deployment becomes unassigned

  lutter

  committed
  cc1259e

• node, store: Add configuration and a separate pool for fdw connections

  lutter

  committed
  85d74f1

• store: Use the fdw pool for copy connections

  lutter

  committed
  56a5f70

• store: Do not copy dynamic data sources more than once

  lutter

  committed
  a2bfd73

• store: Refuse to copy a deployment in a few edge cases

  lutter

  committed
  bca96e1

• store: Fix the mapping of timestamp columns in the copy tables

  lutter

  committed
  7c53888

• store: Reset copy_table_state.started_at when we start working on a table

  lutter

  committed
  3f7f397

• node: Add graphman copy list/status

  lutter

  committed
  e4dab8e

• store: Move advisory locks for migrations to separate module

  lutter

  committed
  54276ee

• store: Change the strategy how copying manages connections

  Show description for ce22d7c
  lutter

  committed
  ce22d7c

• store: Guard against a new and a terminated process copying in parallel

  lutter

  committed
  f2094cf

• store: Fix duration tracking during copying

  lutter

  committed
  3ce547d

• node: Adapt graphman to possibly having multiple deployments

  Show description for 8e91522
  lutter

  committed
  8e91522

• store: Make logs when we wait for a copy conn less alarming

  lutter

  committed
  c0fdde7

• store: Return an error from start_subgraph if copy was canceled

  Show description for 48ee9ad
  lutter

  committed
  48ee9ad

• node: Make 'graphman copy list' more useful

  lutter

  committed
  c6a4e01

• store: Tighten up how WritableStore uses SubgraphStore

  Show description for f1c0458
  lutter

  committed
  f1c0458

• store: Only report active deployments in status

  lutter

  committed
  e307d36

• core: Start subgraphs in the background; copy before loading dds

  Show description for 8cb9b2b
  lutter

  committed
  8cb9b2b

• core, graph: Add the deployment id to internal log messages

  Show description for fdf0a69
  lutter

  committed
  fdf0a69

• core, graph, store: Address review comments

  lutter

  committed
  39935fd

• store: Extract common logic from primary::allocate_site/copy_site

  lutter

  committed
  9388c42

• graph, node, store: Address more review comments

  lutter

  committed
  49631e5

• store: Copy (non-fatal) errors to the new subgraph

  lutter

  committed
  4745e2c

• store: Make sure that copied errors have a unique id

  lutter

  committed
  01c5b8d