Merge master (#18)

* protocols/gossipsub: Fix inconsistency in mesh peer tracking (libp2p#2189) Co-authored-by: Age Manning <[email protected]> * misc/metrics: Add auxiliary crate to record events as OpenMetrics (libp2p#2063) This commit adds an auxiliary crate recording protocol and Swarm events and exposing them as metrics in the OpenMetrics format. * README: Mention [email protected] * examples/: Add file sharing example (libp2p#2186) Basic file sharing application with peers either providing or locating and getting files by name. While obviously showcasing how to build a basic file sharing application, the actual goal of this example is **to show how to integrate rust-libp2p into a larger application**. Architectural properties - Clean clonable async/await interface ([`Client`]) to interact with the network layer. - Single task driving the network layer, no locks required. * examples/README: Give an overview over the many examples (libp2p#2194) * protocols/kad: Enable filtering of (provider) records (libp2p#2163) Introduce `KademliaStoreInserts` option, which allows to filter records. Co-authored-by: Max Inden <[email protected]> * swarm/src/protocols_handler: Impl ProtocolsHandler on either::Either (libp2p#2192) Implement ProtocolsHandler on either::Either representing either of two ProtocolsHandler implementations. Co-authored-by: Thomas Eizinger <[email protected]> * *: Make libp2p-core default features optional (libp2p#2181) Co-authored-by: Max Inden <[email protected]> * core/: Remove DisconnectedPeer::set_connected and Pool::add (libp2p#2195) This logic seems to be a leftover of libp2p#889 and unused today. * protocols/gossipsub: Use ed25519 in tests (libp2p#2197) With f2905c0 the secp256k1 feature is disabled by default. Instead of enabling it in the dev-dependency, simply use ed25519. * build(deps): Update minicbor requirement from 0.10 to 0.11 (libp2p#2200) Updates the requirements on [minicbor](https://gitlab.com/twittner/minicbor) to permit the latest version. - [Release notes](https://gitlab.com/twittner/minicbor/tags) - [Changelog](https://gitlab.com/twittner/minicbor/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/twittner/minicbor/compare/minicbor-v0.10.0...minicbor-v0.11.0) --- updated-dependencies: - dependency-name: minicbor dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Update salsa20 requirement from 0.8 to 0.9 (libp2p#2206) * build(deps): Update salsa20 requirement from 0.8 to 0.9 Updates the requirements on [salsa20](https://github.com/RustCrypto/stream-ciphers) to permit the latest version. - [Release notes](https://github.com/RustCrypto/stream-ciphers/releases) - [Commits](RustCrypto/stream-ciphers@ctr-v0.8.0...salsa20-v0.9.0) --- updated-dependencies: - dependency-name: salsa20 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * *: Bump pnet to v0.22 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <[email protected]> * *: Dial with handler and return handler on error and closed (libp2p#2191) Require `NetworkBehaviourAction::{DialPeer,DialAddress}` to contain a `ProtocolsHandler`. This allows a behaviour to attach custom state to its handler. The behaviour would no longer need to track this state separately during connection establishment, thus reducing state required in a behaviour. E.g. in the case of `libp2p-kad` the behaviour can include a `GetRecord` request in its handler, or e.g. in the case of `libp2p-request-response` the behaviour can include the first request in the handler. Return `ProtocolsHandler` on connection error and close. This allows a behaviour to extract its custom state previously included in the handler on connection failure and connection closing. E.g. in the case of `libp2p-kad` the behaviour could extract the attached `GetRecord` from the handler of the failed connection and then start another connection attempt with a new handler with the same `GetRecord` or bubble up an error to the user. Co-authored-by: Thomas Eizinger <[email protected]> * core/: Remove deprecated read/write functions (libp2p#2213) Co-authored-by: Max Inden <[email protected]> * protocols/ping: Revise naming of symbols (libp2p#2215) Co-authored-by: Max Inden <[email protected]> * protocols/rendezvous: Implement protocol (libp2p#2107) Implement the libp2p rendezvous protocol. > A lightweight mechanism for generalized peer discovery. It can be used for bootstrap purposes, real time peer discovery, application specific routing, and so on. Co-authored-by: rishflab <[email protected]> Co-authored-by: Daniel Karzel <[email protected]> * core/src/network/event.rs: Fix typo (libp2p#2218) * protocols/mdns: Do not fire all timers at the same time. (libp2p#2212) Co-authored-by: Max Inden <[email protected]> * misc/metrics/src/kad: Set query_duration lowest bucket to 0.1 sec (libp2p#2219) Probability for a Kademlia query to return in less than 100 milliseconds is low, thus increasing the lower bucket to improve accuracy within the higher ranges. * misc/metrics/src/swarm: Expose role on connections_closed (libp2p#2220) Expose whether closed connection was a Dialer or Listener. * .github/workflows/ci.yml: Use clang 11 (libp2p#2233) * protocols/rendezvous: Update prost (libp2p#2226) Co-authored-by: Max Inden <[email protected]> * *: Fix clippy warnings (libp2p#2227) * swarm-derive/: Make event_process = false the default (libp2p#2214) Co-authored-by: Max Inden <[email protected]> Co-authored-by: Max Inden <[email protected]> Co-authored-by: Age Manning <[email protected]> Co-authored-by: Ruben De Smet <[email protected]> Co-authored-by: Thomas Eizinger <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: rishflab <[email protected]> Co-authored-by: Daniel Karzel <[email protected]> Co-authored-by: David Craven <[email protected]>
dvc94ch · Sep 14, 2021 · aee258c · aee258c
1 parent ee7a7f5
commit aee258c
Show file tree

Hide file tree

Showing 146 changed files with 8,475 additions and 1,128 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -46,7 +46,7 @@ jobs:
     container:
       image: rust
       env:
-        CC: clang-10
+        CC: clang-11
     steps:
 
     - name: Cancel Previous Runs
@@ -75,7 +75,7 @@ jobs:
         wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
         echo "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-10 main" >> /etc/apt/sources.list
         apt-get update
-        apt-get install -y clang-10
+        apt-get install -y clang-11
 
     - name: Install CMake
       run: apt-get install -y cmake

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,7 @@
 - [`libp2p-ping` CHANGELOG](protocols/ping/CHANGELOG.md)
 - [`libp2p-relay` CHANGELOG](protocols/relay/CHANGELOG.md)
 - [`libp2p-request-response` CHANGELOG](protocols/request-response/CHANGELOG.md)
+- [`libp2p-rendezvous` CHANGELOG](protocols/rendezvous/CHANGELOG.md)
 
 ## Transport Protocols & Upgrades
 
@@ -36,6 +37,7 @@
 
 ## Utilities
 
+- [`libp2p-metrics` CHANGELOG](misc/metrics/CHANGELOG.md)
 - [`multistream-select` CHANGELOG](misc/multistream-select/CHANGELOG.md)
 
 # `libp2p` facade crate
@@ -66,7 +68,9 @@
 
 - Re-export the `wasm-bindgen` feature from `parking_lot`, so
   `libp2p` users can opt-in to that crate's Wasm support. See [PR 2180].
-
+
+- Add `libp2p-metrics`.
+
 [PR 2180]: https://github.com/libp2p/rust-libp2p/pull/2180/
 
 ## Version 0.39.1 [2021-07-12]

diff --git a/Cargo.toml b/Cargo.toml
@@ -26,6 +26,7 @@ default = [
     "quic",
     "relay",
     "request-response",
+    "rendezvous",
     "secp256k1",
     "tcp-async-io",
     "uds",
@@ -38,17 +39,19 @@ dns-async-std = ["libp2p-dns", "libp2p-dns/async-std"]
 dns-tokio = ["libp2p-dns", "libp2p-dns/tokio"]
 floodsub = ["libp2p-floodsub"]
 gossipsub = ["libp2p-gossipsub"]
-identify = ["libp2p-identify"]
-kad = ["libp2p-kad"]
+identify = ["libp2p-identify", "libp2p-metrics/identify"]
+kad = ["libp2p-kad", "libp2p-metrics/kad"]
+metrics = ["libp2p-metrics"]
 mdns = ["libp2p-mdns"]
 mplex = ["libp2p-mplex"]
 noise = ["libp2p-noise"]
-ping = ["libp2p-ping"]
+ping = ["libp2p-ping", "libp2p-metrics/ping"]
 plaintext = ["libp2p-plaintext"]
 pnet = ["libp2p-pnet"]
 quic = ["libp2p-quic"]
 relay = ["libp2p-relay"]
 request-response = ["libp2p-request-response"]
+rendezvous = ["libp2p-rendezvous"]
 tcp-async-io = ["libp2p-tcp", "libp2p-tcp/async-io"]
 tcp-tokio = ["libp2p-tcp", "libp2p-tcp/tokio"]
 uds = ["libp2p-uds"]
@@ -72,15 +75,17 @@ libp2p-floodsub = { version = "0.31.0", path = "protocols/floodsub", optional =
 libp2p-gossipsub = { version = "0.33.0", path = "./protocols/gossipsub", optional = true }
 libp2p-identify = { version = "0.31.0", path = "protocols/identify", optional = true }
 libp2p-kad = { version = "0.32.0", path = "protocols/kad", optional = true }
+libp2p-metrics = { version = "0.1.0", path = "misc/metrics", optional = true }
 libp2p-mplex = { version = "0.30.0", path = "muxers/mplex", optional = true }
 libp2p-noise = { version = "0.33.0", path = "transports/noise", optional = true }
 libp2p-ping = { version = "0.31.0", path = "protocols/ping", optional = true }
 libp2p-plaintext = { version = "0.30.0", path = "transports/plaintext", optional = true }
-libp2p-pnet = { version = "0.21.0", path = "transports/pnet", optional = true }
+libp2p-pnet = { version = "0.22.0", path = "transports/pnet", optional = true }
 libp2p-relay = { version = "0.4.0", path = "protocols/relay", optional = true }
+libp2p-rendezvous = { version = "0.1.0", path = "protocols/rendezvous", optional = true }
 libp2p-request-response = { version = "0.13.0", path = "protocols/request-response", optional = true }
 libp2p-swarm = { version = "0.31.0", path = "swarm" }
-libp2p-swarm-derive = { version = "0.24.0", path = "swarm-derive" }
+libp2p-swarm-derive = { version = "0.25.0", path = "swarm-derive" }
 libp2p-uds = { version = "0.30.0", path = "transports/uds", optional = true }
 libp2p-wasm-ext = { version = "0.30.0", path = "transports/wasm-ext", default-features = false, optional = true }
 libp2p-yamux = { version = "0.34.0", path = "muxers/yamux", optional = true }
@@ -100,19 +105,23 @@ libp2p-websocket = { version = "0.31.0", path = "transports/websocket", optional
 
 [dev-dependencies]
 async-std = { version = "1.6.2", features = ["attributes"] }
+async-trait = "0.1"
 env_logger = "0.9.0"
+structopt = "0.3.21"
 tokio = { version = "1.0.1", features = ["io-util", "io-std", "macros", "rt", "rt-multi-thread"] }
 
 [workspace]
 resolver = "2"
 members = [
     "core",
+    "misc/metrics",
     "misc/multistream-select",
     "misc/peer-id-generator",
     "muxers/mplex",
     "muxers/yamux",
     "protocols/floodsub",
     "protocols/gossipsub",
+    "protocols/rendezvous",
     "protocols/identify",
     "protocols/kad",
     "protocols/mdns",

diff --git a/README.md b/README.md
@@ -12,7 +12,10 @@ This repository is the central place for Rust development of the [libp2p](https:
 - The **[examples](examples)** folder contains small binaries showcasing the
   many protocols in this repository.
 
-- To **report bugs, suggest improvements or request new features** please open
+- For **security related issues** please reach out to [email protected]. Please
+  do not file a public issue on GitHub.
+
+- To **report bugs, suggest improvements or request new features** please open a
   GitHub issue on this repository.
 
 - For **rust-libp2p specific questions** please use the GitHub _Discussions_

diff --git a/core/CHANGELOG.md b/core/CHANGELOG.md
@@ -20,10 +20,32 @@
 - Require `ConnectionHandler::{InEvent,OutEvent,Error}` to implement `Debug`
   (see [PR 2183]).
 
+- Remove `DisconnectedPeer::set_connected` and `Pool::add` (see [PR 2195]).
+
+- Report `ConnectionLimit` error through `ConnectionError` and thus through
+  `NetworkEvent::ConnectionClosed` instead of previously through
+  `PendingConnectionError` and thus `NetworkEvent::{IncomingConnectionError,
+  DialError}` (see [PR 2191]).
+
+- Report abortion of pending connection through `DialError`,
+  `UnknownPeerDialError` or `IncomingConnectionError` (see [PR 2191]).
+
+- Remove deprecated functions `upgrade::write_one`, `upgrade::write_with_len_prefix`
+  and `upgrade::read_one` (see [PR 2213]).
+
+- Add `SignedEnvelope` and `PeerRecord` according to [RFC0002] and [RFC0003]
+  (see [PR 2107]).
+
 [PR 2145]: https://github.com/libp2p/rust-libp2p/pull/2145
+[PR 2213]: https://github.com/libp2p/rust-libp2p/pull/2213
 [PR 2142]: https://github.com/libp2p/rust-libp2p/pull/2142
 [PR 2137]: https://github.com/libp2p/rust-libp2p/pull/2137
 [PR 2183]: https://github.com/libp2p/rust-libp2p/pull/2183
+[PR 2191]: https://github.com/libp2p/rust-libp2p/pull/2191
+[PR 2195]: https://github.com/libp2p/rust-libp2p/pull/2195
+[PR 2107]: https://github.com/libp2p/rust-libp2p/pull/2107
+[RFC0002]: https://github.com/libp2p/specs/blob/master/RFC/0002-signed-envelopes.md
+[RFC0003]: https://github.com/libp2p/specs/blob/master/RFC/0003-routing-records.md
 
 # 0.29.0 [2021-07-12]
 

diff --git a/core/build.rs b/core/build.rs
@@ -19,5 +19,13 @@
 // DEALINGS IN THE SOFTWARE.
 
 fn main() {
-    prost_build::compile_protos(&["src/keys.proto"], &["src"]).unwrap();
+    prost_build::compile_protos(
+        &[
+            "src/keys.proto",
+            "src/envelope.proto",
+            "src/peer_record.proto",
+        ],
+        &["src"],
+    )
+    .unwrap();
 }
diff --git a/core/src/connection.rs b/core/src/connection.rs
@@ -229,10 +229,10 @@ where
         self.handler.inject_event(event);
     }
 
-    /// Begins an orderly shutdown of the connection, returning a
-    /// `Future` that resolves when connection shutdown is complete.
-    pub fn close(self) -> Close<TMuxer> {
-        self.muxing.close().0
+    /// Begins an orderly shutdown of the connection, returning the connection
+    /// handler and a `Future` that resolves when connection shutdown is complete.
+    pub fn close(self) -> (THandler, Close<TMuxer>) {
+        (self.handler, self.muxing.close().0)
     }
 
     /// Polls the connection for events produced by the associated handler

diff --git a/core/src/connection/error.rs b/core/src/connection/error.rs
@@ -29,6 +29,10 @@ pub enum ConnectionError<THandlerErr> {
     // TODO: Eventually this should also be a custom error?
     IO(io::Error),
 
+    /// The connection was dropped because the connection limit
+    /// for a peer has been reached.
+    ConnectionLimit(ConnectionLimit),
+
     /// The connection handler produced an error.
     Handler(THandlerErr),
 }
@@ -41,6 +45,9 @@ where
         match self {
             ConnectionError::IO(err) => write!(f, "Connection error: I/O error: {}", err),
             ConnectionError::Handler(err) => write!(f, "Connection error: Handler error: {}", err),
+            ConnectionError::ConnectionLimit(l) => {
+                write!(f, "Connection error: Connection limit: {}.", l)
+            }
         }
     }
 }
@@ -53,6 +60,7 @@ where
         match self {
             ConnectionError::IO(err) => Some(err),
             ConnectionError::Handler(err) => Some(err),
+            ConnectionError::ConnectionLimit(..) => None,
         }
     }
 }
@@ -63,14 +71,13 @@ pub enum PendingConnectionError<TTransErr> {
     /// An error occurred while negotiating the transport protocol(s).
     Transport(TransportError<TTransErr>),
 
+    /// Pending connection attempt has been aborted.
+    Aborted,
+
     /// The peer identity obtained on the connection did not
     /// match the one that was expected or is otherwise invalid.
     InvalidPeerId,
 
-    /// The connection was dropped because the connection limit
-    /// for a peer has been reached.
-    ConnectionLimit(ConnectionLimit),
-
     /// An I/O error occurred on the connection.
     // TODO: Eventually this should also be a custom error?
     IO(io::Error),
@@ -83,15 +90,13 @@ where
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         match self {
             PendingConnectionError::IO(err) => write!(f, "Pending connection: I/O error: {}", err),
+            PendingConnectionError::Aborted => write!(f, "Pending connection: Aborted."),
             PendingConnectionError::Transport(err) => {
                 write!(f, "Pending connection: Transport error: {}", err)
             }
             PendingConnectionError::InvalidPeerId => {
                 write!(f, "Pending connection: Invalid peer ID.")
             }
-            PendingConnectionError::ConnectionLimit(l) => {
-                write!(f, "Connection error: Connection limit: {}.", l)
-            }
         }
     }
 }
@@ -105,7 +110,7 @@ where
             PendingConnectionError::IO(err) => Some(err),
             PendingConnectionError::Transport(err) => Some(err),
             PendingConnectionError::InvalidPeerId => None,
-            PendingConnectionError::ConnectionLimit(..) => None,
+            PendingConnectionError::Aborted => None,
         }
     }
 }
diff --git a/core/src/connection/manager.rs b/core/src/connection/manager.rs
@@ -20,7 +20,7 @@
 
 use super::{
     handler::{THandlerError, THandlerInEvent, THandlerOutEvent},
-    Connected, ConnectedPoint, Connection, ConnectionError, ConnectionHandler,
+    Connected, ConnectedPoint, ConnectionError, ConnectionHandler, ConnectionLimit,
     IntoConnectionHandler, PendingConnectionError, Substream,
 };
 use crate::{muxing::StreamMuxer, Executor};
@@ -192,6 +192,7 @@ pub enum Event<'a, H: IntoConnectionHandler, TE> {
         /// The error that occurred, if any. If `None`, the connection
         /// has been actively closed.
         error: Option<ConnectionError<THandlerError<H>>>,
+        handler: H::Handler,
     },
 
     /// A connection has been established.
@@ -276,40 +277,6 @@ impl<H: IntoConnectionHandler, TE> Manager<H, TE> {
         ConnectionId(task_id)
     }
 
-    /// Adds an existing connection to the manager.
-    pub fn add<M>(&mut self, conn: Connection<M, H::Handler>, info: Connected) -> ConnectionId
-    where
-        H: IntoConnectionHandler + Send + 'static,
-        H::Handler: ConnectionHandler<Substream = Substream<M>> + Send + 'static,
-        <H::Handler as ConnectionHandler>::OutboundOpenInfo: Send + 'static,
-        TE: error::Error + Send + 'static,
-        M: StreamMuxer + Send + Sync + 'static,
-        M::OutboundSubstream: Send + 'static,
-    {
-        let task_id = self.next_task_id;
-        self.next_task_id.0 += 1;
-
-        let (tx, rx) = mpsc::channel(self.task_command_buffer_size);
-        self.tasks.insert(
-            task_id,
-            TaskInfo {
-                sender: tx,
-                state: TaskState::Established(info),
-            },
-        );
-
-        let task: Pin<Box<Task<Pin<Box<future::Pending<_>>>, _, _, _>>> =
-            Box::pin(Task::established(task_id, self.events_tx.clone(), rx, conn));
-
-        if let Some(executor) = &mut self.executor {
-            executor.exec(task);
-        } else {
-            self.local_spawns.push(task);
-        }
-
-        ConnectionId(task_id)
-    }
-
     /// Gets an entry for a managed connection, if it exists.
     pub fn entry(&mut self, id: ConnectionId) -> Option<Entry<'_, THandlerInEvent<H>>> {
         if let hash_map::Entry::Occupied(task) = self.tasks.entry(id.0) {
@@ -384,14 +351,15 @@ impl<H: IntoConnectionHandler, TE> Manager<H, TE> {
                         new_endpoint: new,
                     }
                 }
-                task::Event::Closed { id, error } => {
+                task::Event::Closed { id, error, handler } => {
                     let id = ConnectionId(id);
                     let task = task.remove();
                     match task.state {
                         TaskState::Established(connected) => Event::ConnectionClosed {
                             id,
                             connected,
                             error,
+                            handler,
                         },
                         TaskState::Pending => unreachable!(
                             "`Event::Closed` implies (2) occurred on that task and thus (3)."
@@ -471,15 +439,15 @@ impl<'a, I> EstablishedEntry<'a, I> {
     ///
     /// When the connection is ultimately closed, [`Event::ConnectionClosed`]
     /// is emitted by [`Manager::poll`].
-    pub fn start_close(mut self) {
+    pub fn start_close(mut self, error: Option<ConnectionLimit>) {
         // Clone the sender so that we are guaranteed to have
         // capacity for the close command (every sender gets a slot).
         match self
             .task
             .get_mut()
             .sender
             .clone()
-            .try_send(task::Command::Close)
+            .try_send(task::Command::Close(error))
         {
             Ok(()) => {}
             Err(e) => assert!(e.is_disconnected(), "No capacity for close command."),
@@ -494,17 +462,6 @@ impl<'a, I> EstablishedEntry<'a, I> {
         }
     }
 
-    /// Instantly removes the entry from the manager, dropping
-    /// the command channel to the background task of the connection,
-    /// which will thus drop the connection asap without an orderly
-    /// close or emitting another event.
-    pub fn remove(self) -> Connected {
-        match self.task.remove().state {
-            TaskState::Established(c) => c,
-            TaskState::Pending => unreachable!("By Entry::new()"),
-        }
-    }
-
     /// Returns the connection ID.
     pub fn id(&self) -> ConnectionId {
         ConnectionId(*self.task.key())