Integrate litep2p into Polkadot SDK

[altonen]

litep2p is a libp2p-compatible P2P networking library. It supports all of the features of rust-libp2p that are currently being utilized by Polkadot SDK.

Compared to rust-libp2p, litep2p has a quite different architecture which is why the new litep2p network backend is only able to use a little of the existing code in sc-network. The design has been mainly influenced by how we'd wish to structure our networking-related code in Polkadot SDK: independent higher-levels protocols directly communicating with the network over links that support bidirectional backpressure. A good example would be NotificationHandle/RequestResponseHandle abstractions which allow, e.g., SyncingEngine to directly communicate with peers to announce/request blocks.

I've tried running polkadot --network-backend litep2p with a few different peer configurations and there is a noticeable reduction in networking CPU usage. For high load (--out-peers 200), networking CPU usage goes down from ~110% to ~30% (80 pp) and for normal load (--out-peers 40), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as enabling receive window auto-tuning, integrating Peerset more closely with litep2p or improving memory usage of the WebSocket transport
b) fixing bugs/instabilities that incorrectly cause litep2p to do less work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage of the new networking backend.

This PR consists of three separate changes:

• introduce a generic PeerId (wrapper around Multihash) so that we don't have use NetworkService::PeerId in every part of the code that uses a PeerId
• introduce NetworkBackend trait, implement it for the libp2p network stack and make Polkadot SDK generic over NetworkBackend
• implement NetworkBackend for litep2p

The new library should be considered experimental which is why rust-libp2p will remain as the default option for the time being. This PR currently depends on the master branch of litep2p but I'll cut a new release for the library once all review comments have been addresses.

[dmitry-markin]

The PR is already nicely split in three commits as per bullet points, so what about publishing it as three separate PRs to make reviewing easier? Does it make sense from the perspective of LOC in every change (i.e., if 90% of changes are in one of three commits, that likely doesn't make sense)?

In the end it's up to you to decide, because rebasing/merging the ladder of three PRs could be quite tedious.

[altonen]

Is there a benefit from splitting the code into three PRs compared to reviewing the three commits separately in this PR? The workload for the review is the same as far as I see it.

[dmitry-markin]

A possible benefit I had in mind is that when the code is updated (i.e., the review comments are addressed), it would be instantly visible in separate PRs, but the individual commits would stay the same, so all 9700 LOC of changes should be viewed to get the understanding of what the code is after additional commits are pushed.

[dmitry-markin]

I'm not pushing for this, as this would come at the cost of rebasing the PRs

[altonen]

Do other people have any opinions? Would you like to see this split into multiple PRs or is it ok to review the three commits separately?

[alindima]

I don't mind reviewing as a single PR

[koute]

Hm... I think actually splitting this PR might be a good idea. Not necessarily because of making it easier to review, but simply because that'd probably speed up how fast we can merge it. The PeerId commit is a good example of something that we could just merge almost immediately.

[altonen]

Hm... I think actually splitting this PR might be a good idea. Not necessarily because of making it easier to review, but simply because that'd probably speed up how fast we can merge it. The PeerId commit is a good example of something that we could just merge almost immediately.

I don't mind if the review takes a bit longer because honestly I see now value in getting the first two commits into master while the third one is still in review.

[alindima]

Outstanding work! 🚀

[dmitry-markin]

Some comments (mostly nits) for the sc_network_types::PeerId commit (the first commit).

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/paritytech-update-for-april/7646/1

[kianenigma]

I understand that this is an important change, and a technological achievement indeed (all judging by the number of emoji reaction 🚀), but I want to ask for future similar changes to
be made with possibly less changes to the node side code (service.rs etc), if possible.

Please see #5, https://forum.polkadot.network/t/polkadot-parachain-omni-node-gathering-ideas-and-feedback/7823 and similar :)

[eskimor]

@michalkucharczyk good data point for the improvements we want to make. Networking changes should obviously not affect client code, if they are not customizing things.

[bkchr]

I had started locally to fix this up, but have given up. The reviewers should not have accepted it in this state, especially for an experimental feature.