Implement new RPC API spec

[jsdw]

The Vision

The goal here is to implement Pierre's new RPC API.

The spec is documented here:

https://paritytech.github.io/json-rpc-interface-spec/introduction.html

The repository for said spec:

https://github.com/paritytech/json-rpc-interface-spec/

The Plan

• Implement and merge the chainSpec_* methods. properties and genesisHash are two easy starting points.
  • This will also give us a crate/area to build on in future steps. Let's try and minimise the risk of merge conflicts w.r.t other substrate work.
• Implement and merge transaction_* methods.
• Implement and merge chainHead_* methods.
  • These involve a new concept of "pinning" blocks while a subscription knows about them, and allowing blocks to be unpinned from subscriptions.
  • In general, these methods will be the hardest to implement.
• Implement and merge archive_* methods.
  • These are lowest priority (ie least useful to UIs and such; subxt for instance will rely mostly on the above for instance).
  • Some of these are undocumented. Let's discuss when we get here what can be skipped, but assume undocumented methods in the spec can be skipped for now.

There will be a bunch of back and forth, especially to begin with. Getting the first implementation work done will give us a better idea on the time taken to do the rest.

There will be some opportunity to parallelise this work after the initial crate and structure are in place.

The existing RPC methods will remain in place for a while after this merges, giving people plenty of time to migrate things over.

If we run into any issues understanding the spec, let's raise issues on the repo (linked above) to help make sure it's super clear.

(Thanks @tomaka for your input in helping create this plan!)

[lexnv]

The MVP part of this issue has been reviewed and merged into substrate.

To summarise, we have completed the following:

Rpc API

1. chainSpec rpc: Implement chainSpec RPC API #12261 - inspect the chain specification

Methods included:

• chainSpec_unstable_chainName - Get the chain name, as present in the chain specification.
• chainSpec_unstable_genesisHash - Get the chain's genesis hash.
• chainSpec_unstable_properties - Get the properties of the chain, as present in the chain specification.

2. transaction rpc: Implement transaction RPC API #12328 - submit extrinsics to the chain

This will become the standard way of submitting transactions in the V2 API and
aims to simplify the current V1 and enrich the information provided by the chain.
The PR implies a change in how transactions are reported by the tx-pool to include
further information (the extrinsic index in the block, feature needed by #10406).
Although the TransactionStatus emitted by the tx-pool has changed, this PR avoids
a breaking change and keeps compatibility with the old V1 API over the wire by
implementing a custom serialization scheme.

Methods included:

• transaction_unstable_submitAndWatch - Submit and watch an extrinsic via RPC pubsub
• transaction_unstable_unwatch - Unsubscribe

3. chainHead rpc: Implement chainHead RPC API #12544 - track the head of the chain

This PR enables the seamless integration of light-clients with substrate by
providing a standard way of fetching information about the most recent
blocks in the chain, and executing runtime API calls against them. The PR implements
the RPC Spec that provides a subscription stream of blocks (finalized and best).
Using the data provided by the subscription, the user can then query block's
body / header / storage and make runtime API calls. The substrate guarantees
that the blocks from the canonical chain are available (not pruned/removed)
for a sensible amount of time -- this defines the block's pining.

Methods included:

• chainHead_unstable_follow - Subscription for new, new best and finalized blocks
• chainHead_unstable_body - Get the extrinsics of a reported block by the follow method
• chainHead_unstable_call - Runtime API call at a reported block
• chainHead_unstable_header - Get the header of a reported block
• chainHead_unstable_storage - Query storage for a reported block
• chainHead_unstable_unpin - Unpin a reported block

Blocks Pinning

For noncanonical blocks to remain in memory of the node (be pinned) the following
PR was created #12497. The PR works by delaying the pruning of blocks by a number
of finalization. The PR represents an intermediate way of achieving the blocks pinning
in a reasonable development timeline. And is a good tradeoff to handle the edge case
of blocks being unpinned by the pruning of the database.

The complete substrate API for blocks pinning is design here: #12475 (comment).

We have decided to pin all blocks reported by import_notification_stream and
finality_notification_stream. The reported blocks are reference counted and unpinned
when the last reference is dropped. Blocks are to be kept in memory to avoid
database side-effects. The RPC layer must enforce an appropriate limit
for the pinning of blocks.

@bkchr 's team has offered support to prioritize this #12497 (comment).

Outside of Substrate

JsonRPSee related work:

• server: Expose the subscription ID jsonrpsee#900
• proc macro: support camelCase & snake_case for object params jsonrpsee#921
• chore: release 0.16.1 jsonrpsee#943
• rpc server with HTTP/WS on the same socket #12663

Subxt related work:

• rpc: Expose the subscription ID for RpcClientT subxt#733

To gain feedback and start the adoption of this API, subxt will provide
a trial support paritytech/subxt#732.
There are also coordinated efforts made by @harrysolovay's team to
integrate this API as a first-class citizen in capi.

Next Steps

• Add API for block pinning #12475 (comment) - Add complete API support for blocks pinning
• rpc: Implement archive RPC API #12737 - Merge the archive PR after the design is stabilized
• Extend testing scenarios
• Monitor the V2 API via adding metrics
• Stabilize the unstable methods (after this, any change to the APIs functionality will be an incremental version change).

Thanks everyone for making this happen!

// nonexhaustive CC @tomaka @skunert @bkchr @arkpar @cheme @jsdw @niklasad1