New gossip_net backend using libp2p (#3988)

This is the revival of the tmp/cmr/net2 branch rebased onto develop. Some important user-facing changes: - No separate discovery/communication/etc ports. One port for all public daemon communications. - Automatic port forwarding with UPnP. If your local network supports UPnP, there should be no configuration required. - Local peer discovery. If your local network supports mDNS broadcast, coda daemons will automatically discover each other. This includes several daemons on the same machine- no more building peer lists! - New libp2p keypairs. These are managed the same as our key pairs with secret_file. Without configuration, key pairs are ephemeral and will disappear when the daemon restarts. (TODO: should we instead persist the keypair? does it matter for non-infrastructure?) Some important internal changes: - All daemon-daemon connections are now authenticated and confidential. - Connections are no longer transient and per-request. Individual requests get multiplexed as their own stream over the one connection between the peers. This is analogous to HTTP/2. Outgoing connections will appear to originate from the libp2p listening port, vs some transient port. Outstanding details: - Trust system needs to get augmented to track Peer.t instead of just an IP. Until then we can't implement ban_notify (#4093, #4096). - Libp2p has little per-connection structured reporting, some things we currently penalize trust for are not detected (eg opening a libp2p connection without also opening a coda RPC stream) (#4098). - New pubsub allows banning senders by peer ID. We currently don't do this but we should ban peerIDs that originated bad info and not just the IP of the whoever relayed it to us (#4096). - ~~Current pubsub validation flow goes a bit against the libp2p grain, and it's not clear to me that the current behavior will survive [this libp2p PR](libp2p/go-libp2p-kad-dht#388). There's an inline comment near the should_forward_message impl (#4097).~~ done - Connection limit enforcement (#4095) Other changes: - Rips out the last vestiges of old membership, which aren't in use. - The connection info in envelopes is much more accurate now. We shouldn't start trusting it just yet due to some future vagaries around relaying. - bump nixpkgs version Future improvements: - IPv6. There's a hardcoded IPv4 assumption in the helper around IP filtering. - Investigate libp2p autorelay. This should help nodes in restrictive networks achieve better connectivity, but has a host of problems. - Intelligent request routing. I believe we can use the "provider" feature to, at the very least, only send eg sync/bootstrap requests to nodes who believe themselves to be in sync. There are other options.
MinaProtocol · Feb 10, 2020 · 16b8125 · 16b8125
1 parent 79a007b
commit 16b8125
Show file tree

Hide file tree

Showing 169 changed files with 4,779 additions and 43,210 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
diff --git a/.circleci/config.yml.jinja b/.circleci/config.yml.jinja
@@ -52,7 +52,7 @@ jobs:
     build-archive:
         resource_class: xlarge
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
               environment:
                 CODA_DOCKER: true
                 HASURA_PORT: 8080
@@ -98,7 +98,7 @@ jobs:
                   command: ./scripts/archive/build-release-archives.sh
     lint:
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
             - run:
@@ -127,9 +127,12 @@ jobs:
                   command: ./scripts/require-ppx-coda.py
     lint-opt:
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
+            - run:
+                name: Show USER
+                command: echo $USER
             - run:
                   name: Update Submodules
                   command: git submodule sync && git submodule update --init --recursive
@@ -151,59 +154,6 @@ jobs:
                   name: Update branch protection rule from test configuration
                   command: ./scripts/test.py required-status >required_status && cat required_status && ./scripts/update_branch_rule.py required_status
 
-    build-auxiliary:
-        docker:
-            - image: nixos/nix
-        steps:
-            - checkout
-            - run:
-                name: Install tools
-                command: apk add patchelf dpkg tar
-            - run:
-                name: Install cachix
-                command: nix-env -iA cachix -f https://cachix.org/api/v1/install
-            - run:
-                name: Build libp2p_helper using cachix
-                command: |
-                    cachix use codaprotocol
-                    cd src/app/libp2p_helper
-
-                    if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
-                        echo "CACHIX_SIGNING_KEY is not set"
-                        nix-build default.nix
-                    else
-                        nix-build default.nix | cachix push codaprotocol
-                    fi
-            - run:
-                name: Create package dir
-                command: mkdir -p package
-            - run:
-                name: Collect and strip binary
-                command: |
-                    cp src/app/libp2p_helper/result/bin/libp2p_helper package/coda-libp2p_helper
-                    chmod +w package/coda-libp2p_helper
-                    patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 package/coda-libp2p_helper
-            - run:
-                name: Create deb
-                command: |
-                    mkdir -p package/deb/usr/local/sbin
-                    cp package/coda-libp2p_helper package/deb/usr/local/sbin/.
-                    mkdir -p package/deb/DEBIAN
-                    cp src/app/libp2p_helper/build/DEBIAN/control package/deb/DEBIAN/control
-                    DATE=$(date +%Y-%m-%d)
-                    sed -i "s/DATE/${DATE}/" package/deb/DEBIAN/control
-                    find src/app/libp2p_helper -type f -print0  | xargs -0 sha1sum | sort | sha1sum | cut -f 1 -d ' ' > /tmp/sha1sig
-                    HASH=$(cat /tmp/sha1sig)
-                    sed -i "s/HASH/${HASH}/" package/deb/DEBIAN/control
-                    echo "------------------------------------"
-                    cat package/deb/DEBIAN/control
-                    echo "------------------------------------"
-                    find package/deb/
-                    echo "------------------------------------"
-                    dpkg-deb -v --build package/deb/ package/coda-discovery-${HASH}.deb
-            - store_artifacts:
-                  path: package
-
     build-macos:
         macos:
             xcode: "10.2.1"
@@ -287,9 +237,9 @@ jobs:
                     cd src/app/libp2p_helper
                     if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
                         echo "CACHIX_SIGNING_KEY is not set"
-                        nix-build default.nix
+                        nix-build --option sandbox false default.nix
                     else
-                        nix-build default.nix | cachix push codaprotocol
+                        nix-build --option sandbox false default.nix | cachix push codaprotocol
                     fi
 
             ### dune
@@ -353,7 +303,7 @@ jobs:
     build-artifacts--{{profile}}:
         resource_class: xlarge
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
             - run:
@@ -366,6 +316,22 @@ jobs:
             - run:
                   name: Pin external packages
                   command: ./scripts/pin-external-packages.sh
+            - run:
+                name: Install cachix
+                command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
+            - run:
+                name: Build libp2p_helper using cachix
+                command: |
+                    export USER=opam 
+                    . ~/.nix-profile/etc/profile.d/nix.sh
+                    cachix use codaprotocol
+                    cd src/app/libp2p_helper
+                    if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
+                        echo "CACHIX_SIGNING_KEY is not set"
+                        nix-build --option sandbox false default.nix
+                    else
+                        nix-build --option sandbox false default.nix | cachix push codaprotocol
+                    fi
             - run:
                   name: Build OCaml
                   command: ./scripts/skip_if_only_frontend.sh bash -c 'eval `opam config env` && make build 2>&1 | tee /tmp/artifacts/buildocaml.log'
@@ -434,7 +400,7 @@ jobs:
     test-unit--{{profile}}:
         resource_class: xlarge
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
             - run: ulimit -c unlimited
@@ -463,6 +429,22 @@ jobs:
                     #chmod -R 777 src/app/libp2p_helper
                      #Do the build
                     #sudo -H -u nix bash -c '. ~/.nix-profile/etc/profile.d/nix.sh && cachix use codaprotocol && cd src/app/libp2p_helper && nix-build default.nix'
+            - run:
+                name: Install cachix
+                command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
+            - run:
+                name: Build libp2p_helper using cachix
+                command: |
+                    export USER=opam
+                    . ~/.nix-profile/etc/profile.d/nix.sh
+                    cachix use codaprotocol
+                    cd src/app/libp2p_helper
+                    if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
+                        echo "CACHIX_SIGNING_KEY is not set"
+                        nix-build --option sandbox false default.nix
+                    else
+                        nix-build --option sandbox false default.nix | cachix push codaprotocol
+                    fi
             - run:
                   name: Run unit tests
                   command: ./scripts/skip_if_only_frontend.sh bash -c 'source ~/.profile && dune build --profile={{profile}} -j8 && (dune runtest src/lib --profile={{profile}} -j8 || (./scripts/link-coredumps.sh && false))'
@@ -475,13 +457,29 @@ jobs:
     test-unit--{{profile}}:
         resource_class: xlarge
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
             - run: ulimit -c unlimited
             - run:
                   name: Update OPAM
                   command: ./scripts/update-opam-in-docker.sh
+            - run:
+                name: Install cachix
+                command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
+            - run:
+                name: Build libp2p_helper using cachix
+                command: |
+                    export USER=opam
+                    . ~/.nix-profile/etc/profile.d/nix.sh
+                    cachix use codaprotocol
+                    cd src/app/libp2p_helper
+                    if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
+                        echo "CACHIX_SIGNING_KEY is not set"
+                        nix-build --option sandbox false default.nix
+                    else
+                        nix-build --option sandbox false default.nix | cachix push codaprotocol
+                    fi
             - run:
                   name: Run unit tests
                   command: ./scripts/skip_if_only_frontend.sh bash -c 'source ~/.profile && dune build --profile={{profile}} -j8 && (dune runtest src/lib --profile={{profile}} -j8 || (./scripts/link-coredumps.sh && false))'
@@ -494,12 +492,28 @@ jobs:
     test--{{profile}}:
         resource_class: large
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
             - run:
                   name: Update OPAM
                   command: ./scripts/update-opam-in-docker.sh
+            - run:
+                name: Install cachix
+                command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
+            - run:
+                name: Build libp2p_helper using cachix
+                command: |
+                    export USER=opam
+                    . ~/.nix-profile/etc/profile.d/nix.sh
+                    cachix use codaprotocol
+                    cd src/app/libp2p_helper
+                    if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
+                        echo "CACHIX_SIGNING_KEY is not set"
+                        nix-build --option sandbox false default.nix
+                    else
+                        nix-build --option sandbox false default.nix | cachix push codaprotocol
+                    fi
             {%- for test in small_curves_tests[profile] %}
             - run:
                   name: Running test -- {{profile}}:{{test}}
@@ -513,12 +527,28 @@ jobs:
     test--{{profile}}:
         resource_class: xlarge
         docker:
-            - image: codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e
+            - image: codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9
         steps:
             - checkout
             - run:
                   name: Update OPAM
                   command: ./scripts/update-opam-in-docker.sh
+            - run:
+                name: Install cachix
+                command: USER=opam . ~/.nix-profile/etc/profile.d/nix.sh && nix-env --option sandbox false -iA cachix -f https://cachix.org/api/v1/install
+            - run:
+                name: Build libp2p_helper using cachix
+                command: |
+                    export USER=opam
+                    . ~/.nix-profile/etc/profile.d/nix.sh
+                    cachix use codaprotocol
+                    cd src/app/libp2p_helper
+                    if [ -z ${CACHIX_SIGNING_KEY+x} ]; then
+                        echo "CACHIX_SIGNING_KEY is not set"
+                        nix-build --option sandbox false default.nix
+                    else
+                        nix-build --option sandbox false default.nix | cachix push codaprotocol
+                    fi
             {%- for test in medium_curves_and_other_tests[profile] %}
             - run:
                   name: Running test -- {{profile}}:{{test}}
@@ -537,7 +567,6 @@ workflows:
         jobs:
             - lint
             - lint-opt
-            - build-auxiliary
             - update-branch-protection:
                 filters:
                   branches:

diff --git a/.dockerignore b/.dockerignore
@@ -1,7 +1,7 @@
 *
 !src/opam.export
 !coda.deb
+!coda-discovery.deb
 !src/external/**/*
-!src/app/kademlia-haskell
 !src/app/libp2p_helper
 !src/app/trace-tool
diff --git a/.gitattributes b/.gitattributes
@@ -11,6 +11,5 @@ frontend/website/static/presskit.zip filter=lfs diff=lfs merge=lfs -text
 *.woff filter=lfs diff=lfs merge=lfs -text
 *.woff2 filter=lfs diff=lfs merge=lfs -text
 frontend/website/static/*.bc.js filter=lfs diff=lfs merge=lfs -text
-src/app/kademlia-haskell/packages.nix linguist-generated=true
 src/app/archive/archive_graphql_schema.json linguist-generated=true
 docs/res/block_production_fsm.dot.png filter=lfs diff=lfs merge=lfs -text
diff --git a/.github/labels.json b/.github/labels.json
@@ -32,7 +32,7 @@
     "color": "#9791a0"
   },
   {
-    "name": "area-kademlia",
+    "name": "area-libp2p",
     "color": "#9791a0"
   },
   {

diff --git a/.mergify.yml b/.mergify.yml
@@ -2,7 +2,6 @@ pull_request_rules:
     - name: automatically merge approved PRs with the ready-to-merge label
       conditions:
           - "status-success=ci/circleci: build-artifacts--testnet_postake_medium_curves"
-          - "status-success=ci/circleci: build-auxiliary"
           - "status-success=ci/circleci: build-wallet"
           - "status-success=ci/circleci: lint"
           - "status-success=ci/circleci: test--fake_hash"

diff --git a/CODEOWNERS b/CODEOWNERS
@@ -18,7 +18,6 @@
 /src/app/cli/src/tests @emberian @bkase @ghost-not-in-the-shell @deepthiskumar
 /src/app/cli/src/coda.ml @nholland94 @bkase @emberian @imeckler
 /src/app/cli/src/init @nholland94 @bkase @emberian @imeckler
-/src/app/kademlia-haskell/ @bkase @emberian @psteckler
 /src/app/libp2p_helper @emberian @bkase @nholland94
 /src/app/reformat/ @bkase
 /src/app/trace-tool/ @emberian @mrmr1993
@@ -50,7 +49,6 @@
 /src/lib/group_map/ @rbkhmrcr @vanishreerao @imeckler
 /src/lib/hash_prefixes/ @rbkhmrcr @vanishreerao @imeckler
 /src/lib/interruptible/ @bkase @nholland94
-/src/lib/kademlia @bkase @emberian @psteckler
 /src/lib/key_value_database/ @bkase @nholland94 @psteckler
 /src/lib/keys_lib/ @emberian @mrmr1993 @imeckler
 /src/lib/ledger_catchup/ @nholland94 @ghost-not-in-the-shell

diff --git a/Makefile b/Makefile
@@ -8,8 +8,7 @@ GITLONGHASH = $(shell git rev-parse HEAD)
 MYUID = $(shell id -u)
 DOCKERNAME = codabuilder-$(MYUID)
 
-# Unique signature of kademlia code tree
-KADEMLIA_SIG = $(shell cd src/app/kademlia-haskell ; find . -type f -print0  | xargs -0 sha1sum | sort | sha1sum | cut -f 1 -d ' ')
+# Unique signature of libp2p code tree
 LIBP2P_HELPER_SIG = $(shell cd src/app/libp2p_helper ; find . -type f -print0  | xargs -0 sha1sum | sort | sha1sum | cut -f 1 -d ' ')
 
 ifeq ($(DUNE_PROFILE),)
@@ -61,15 +60,9 @@ clean:
 	@rm -rf _build
 	@rm -rf src/$(COVERAGE_DIR)
 
-kademlia:
-	@# FIXME: Bash wrap here is awkward but required to get nix-env
-	bash -c "source ~/.profile && cd src/app/kademlia-haskell && nix-build release2.nix"
-
 libp2p_helper:
 	bash -c "source ~/.profile && cd src/app/libp2p_helper && nix-build default.nix"
 
-# Alias
-dht: kademlia libp2p_helper
 
 GENESIS_DIR := $(TMPDIR)/coda_cache_dir
 
@@ -105,7 +98,7 @@ update-opam:
 macos-portable:
 	@rm -rf _build/coda-daemon-macos/
 	@rm -rf _build/coda-daemon-macos.zip
-	@./scripts/macos-portable.sh src/_build/default/src/app/cli/src/coda.exe src/app/kademlia-haskell/result/bin/kademlia _build/coda-daemon-macos
+	@./scripts/macos-portable.sh src/_build/default/src/app/cli/src/coda.exe _build/coda-daemon-macos
 	@zip -r _build/coda-daemon-macos.zip _build/coda-daemon-macos/
 	@echo Find coda-daemon-macos.zip inside _build/
 
@@ -177,15 +170,6 @@ docker-toolchain-rust:
 		echo "Repo has uncommited changes, commit first to set hash." ;\
 	fi
 
-# All in one step to build toolchain and binary for kademlia
-# TODO: Rename to docker-toolchain-discovery
-docker-toolchain-haskell:
-	@echo "Building codaprotocol/coda:toolchain-haskell-$(KADEMLIA_SIG)" ;\
-    docker build --file dockerfiles/Dockerfile-toolchain-haskell --tag codaprotocol/coda:toolchain-haskell-$(KADEMLIA_SIG) . ;\
-    echo  'Extracting deb package' ;\
-    mkdir -p _build ;\
-    docker run --rm --entrypoint cat codaprotocol/coda:toolchain-haskell-$(KADEMLIA_SIG) /src/coda-discovery.deb > _build/coda-discovery.deb
-
 update-deps:
 	./scripts/update-toolchain-references.sh $(GITLONGHASH)
 	make render-circleci
@@ -316,4 +300,4 @@ ml-docs:
 # unless there is a reason not to.
 # https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html
 # HACK: cat Makefile | egrep '^\w.*' | sed 's/:/ /' | awk '{print $1}' | grep -v myprocs | sort | xargs
-.PHONY: all base-docker base-googlecloud base-minikube build check-format ci-base-docker clean codaslim containerstart deb dev codabuilder kademlia coda-docker coda-googlecloud coda-minikube ocaml407-googlecloud pull-ocaml407-googlecloud reformat test test-all test-coda-block-production-sig test-coda-block-production-stake test-codapeers-sig test-codapeers-stake test-full-sig test-full-stake test-runtest test-transaction-snark-profiler-sig test-transaction-snark-profiler-stake update-deps render-circleci check-render-circleci docker-toolchain-rust toolchains doc_diagrams ml-docs macos-setup macos-setup-download macos-setup-compile
+.PHONY: all base-docker base-googlecloud base-minikube build check-format ci-base-docker clean codaslim containerstart deb dev codabuilder coda-docker coda-googlecloud coda-minikube ocaml407-googlecloud pull-ocaml407-googlecloud reformat test test-all test-coda-block-production-sig test-coda-block-production-stake test-codapeers-sig test-codapeers-stake test-full-sig test-full-stake test-runtest test-transaction-snark-profiler-sig test-transaction-snark-profiler-stake update-deps render-circleci check-render-circleci docker-toolchain-rust toolchains doc_diagrams ml-docs macos-setup macos-setup-download macos-setup-compile libp2p_helper
diff --git a/README-dev.md b/README-dev.md
@@ -59,7 +59,7 @@ of the repo.
 
 * Pull down developer container image  (~2GB download, go stretch your legs)
 
-`docker pull codaprotocol/coda:toolchain-54430467ba429af285ea937d1c1da7d4b4cbde3e`
+`docker pull codaprotocol/coda:toolchain-6728e849d78ccb9ca7c53d1b82609c8dd49b40f9`
 
 * Create local builder image
 
@@ -128,12 +128,12 @@ You should probably use `USEDOCKER=TRUE` unless you've done the [building withou
 
 These are the most important `make` targets:
 
-* `libp2p_helper`: build the libp2p helper
 * `build`: build everything
 * `docker`: build the container
 * `container`: restart the development container (or start it if it's not yet)
 * `dev`: does `docker`, `container`, and `build`
 * `test`: run the tests
+* `libp2p_helper`: build the libp2p helper
 * `web`: build the website, including the state explorer
 
 We use the [dune](https://github.com/ocaml/dune/) buildsystem for our OCaml code.