network: Update libp2p to 0.54.1 #5996
Comments
|
CC @paritytech/networking |
|
It's definitely something we want to do! I would wait a bit to update libp2p to the latest version:
After we update libp2p to latest version, we could start looking at performance issues: #5221 |
|
We jump onto latest version of libp2p at https://github.com/autonomys/subspace fairly quickly and while I agree it is annoying to do all the changes needed to migrate to new version, it typically takes me a couple of hours to do. And I am yet to see show-stopping issues after upgrade itself (though we do not have as robust testing and benchmarking capabilities yet). I can probably do this upgrade for you (in two steps: first to 0.53.x and then to 0.54.x to make review easier), but it takes you guys a really long time to review with regular merge conflicts in between (I have 4 PRs that I opened that are waiting for review for over a week each, some are waiting for way longer than a week), so while I understand you probably have good reasons for this, it is a big investment. Let me know if you're willing to commit to timely review and I'll take a look at doing this in the near future myself, it isn't a big deal. Substrate is a large source of outdated dependencies that triggered RUSTSEC alerts in the past already as well as causing other issues, like pulling old version of |
|
@nazar-pc We would love to get some help on this if you have some extra bandwidth! 🙏 I'll have a pass soon for the remaining networking PRs, thanks for contributing again! |
|
Noted, I'll update once I get to it |
|
0.53.2 is done here: #6248 Please assign this issue to me, I'll take care of the rest too. UPD: 0.54.1 is also there, it was too easy to do to delay it further. |
|
Now https://rustsec.org/advisories/RUSTSEC-2024-0421.html affects us due to old libp2p version as well |
|
@nazar-pc what is the impact of https://rustsec.org/advisories/RUSTSEC-2024-0421.html issue for your team? IIUC, https://rustsec.org/advisories/RUSTSEC-2024-0421.html can lead to privilege escalation if and only if host domain name check is part of a privilege check / authorization check. From the substrate perspective, we use crypto/noise on top of every connection, we verify and validate the remote is in possession of the private keys derived from the PeerID. In other words, even if the DNS resolver decides that Let me know if my understanding is correct 🙏 |
|
Probably not very large, but I really don't like to suppress advisories and I had to do that more than once due to old libp2p and litep2p versions in Substrate already. |
|
Yep totally understandable, thanks for the swift reply 🙏 We'll test until the EoW the libp2p update which should enable us to easily upgrade to the latest unaffected version 👍 |
# Description Fixes #5996 https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0 https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md ## Integration Nothing special is needed, just note that `yamux_window_size` is no longer applicable to libp2p (litep2p seems to still have it though). ## Review Notes There are a few simplifications and improvements done in libp2p 0.53 regarding swarm interface, I'll list a few key/applicable here. libp2p/rust-libp2p#4788 removed `write_length_prefixed` function, so I inlined its code instead. libp2p/rust-libp2p#4120 introduced new `libp2p::SwarmBuilder` instead of now deprecated `libp2p::swarm::SwarmBuilder`, the transition is straightforward and quite ergonomic (can be seen in tests). libp2p/rust-libp2p#4581 is the most annoying change I have seen that basically makes many enums `#[non_exhaustive]`. I mapped some, but those that couldn't be mapped I dealt with by printing log messages once they are hit (the best solution I could come up with, at least with stable Rust). libp2p/rust-libp2p#4306 makes connection close as soon as there are no handler using it, so I had to replace `KeepAlive::Until` with an explicit future that flips internal boolean after timeout, achieving the old behavior, though it should ideally be removed completely at some point. `yamux_window_size` is no longer used by libp2p thanks to libp2p/rust-libp2p#4970 and generally Yamux should have a higher performance now. I have resolved and cleaned up all deprecations related to libp2p except `BandwidthSinks`. Libp2p deprecated it (though it is still present in 0.54.1, which is why I didn't handle it just yet). Ideally Substrate would finally [switch to the official Prometheus client](paritytech/substrate#12699), in which case we'd get metrics for free. Otherwise a bit of code will need to be copy-pasted to maintain current behavior with `BandwidthSinks` gone, which I left a TODO about. The biggest change in 0.54.0 is libp2p/rust-libp2p#4568 that changed transport APIs and enabled unconditional potential port reuse, which can lead to very confusing errors if running two Substrate nodes on the same machine without changing listening port explicitly. Overall nothing scary here, but testing is always appreciated. # Checklist * [x] My PR includes a detailed description as outlined in the "Description" and its two subsections above. * [x] My PR follows the [labeling requirements]( https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process ) of this project (at minimum one label for `T` required) * External contributors: ask maintainers to put the right label on your PR. --- Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd --------- Co-authored-by: Dmitry Markin <[email protected]>
# Description Fixes paritytech#5996 https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0 https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md ## Integration Nothing special is needed, just note that `yamux_window_size` is no longer applicable to libp2p (litep2p seems to still have it though). ## Review Notes There are a few simplifications and improvements done in libp2p 0.53 regarding swarm interface, I'll list a few key/applicable here. libp2p/rust-libp2p#4788 removed `write_length_prefixed` function, so I inlined its code instead. libp2p/rust-libp2p#4120 introduced new `libp2p::SwarmBuilder` instead of now deprecated `libp2p::swarm::SwarmBuilder`, the transition is straightforward and quite ergonomic (can be seen in tests). libp2p/rust-libp2p#4581 is the most annoying change I have seen that basically makes many enums `#[non_exhaustive]`. I mapped some, but those that couldn't be mapped I dealt with by printing log messages once they are hit (the best solution I could come up with, at least with stable Rust). libp2p/rust-libp2p#4306 makes connection close as soon as there are no handler using it, so I had to replace `KeepAlive::Until` with an explicit future that flips internal boolean after timeout, achieving the old behavior, though it should ideally be removed completely at some point. `yamux_window_size` is no longer used by libp2p thanks to libp2p/rust-libp2p#4970 and generally Yamux should have a higher performance now. I have resolved and cleaned up all deprecations related to libp2p except `BandwidthSinks`. Libp2p deprecated it (though it is still present in 0.54.1, which is why I didn't handle it just yet). Ideally Substrate would finally [switch to the official Prometheus client](paritytech/substrate#12699), in which case we'd get metrics for free. Otherwise a bit of code will need to be copy-pasted to maintain current behavior with `BandwidthSinks` gone, which I left a TODO about. The biggest change in 0.54.0 is libp2p/rust-libp2p#4568 that changed transport APIs and enabled unconditional potential port reuse, which can lead to very confusing errors if running two Substrate nodes on the same machine without changing listening port explicitly. Overall nothing scary here, but testing is always appreciated. # Checklist * [x] My PR includes a detailed description as outlined in the "Description" and its two subsections above. * [x] My PR follows the [labeling requirements]( https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process ) of this project (at minimum one label for `T` required) * External contributors: ask maintainers to put the right label on your PR. --- Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd --------- Co-authored-by: Dmitry Markin <[email protected]> (cherry picked from commit c881288)
Substrate currently uses libp2p 0.52.4 with lots of outdated dependencies, libp2p 0.53.0 was released 11 months ago and 0.54.0 was released 2 months ago in time for September's release cut.
This is way too slow, please consider upgrading libp2p to not lag behind for a year or more, this is bad for downstream users due to lots of duplicated and outdated dependencies, can also be problematic from security standpoint.
The text was updated successfully, but these errors were encountered: