p2p: test scenarios support #5962

algorandskiy · 2024-03-18T20:36:42Z

Summary

add P2PBootstrap netdeploy's template option
add p2p dnsadddr support to netdeploy
fix DNSSecurityTXTEnforced use for DNS TXT queries instead of DNSSecuritySRVEnforced
add scenario1s-p2p, hello-word-tiny-p2p, hello-world-small-p2p
fix libp2p and DHT metrics exporters
add missing gossipSubPeer.RoutingAddr implementation needed for TX handler when throttled with ERL
add exponential backoff to p2p meshThread
fix outgoing connection counting, and use conn direction rather than stream direction for incoming/outgoing peer tagging
make libp2 logger to log into node.log by implementing zapcore.Core writing via logger.Logger
Example log records

{
    "file": "github.com/libp2p/[email protected]/config/config.go",
    "function": "github.com/libp2p/go-libp2p/config.(*Config).NewNode",
    "level": "warning",
    "line": 304,
    "msg": "rcmgr limit conflicts with connmgr limit: conn manager high watermark limit: 192, exceeds the system connection limit of: 1",
    "name": "127.0.0.1:0",
    "libp2p": "p2p-config",
    "time": "2024-04-17T19:43:31.640497-04:00"
}
{
    "file": "github.com/libp2p/[email protected]/pubsub.go",
    "function": "github.com/libp2p/go-libp2p-pubsub.(*PubSub).processLoop",
    "level": "info",
    "line": 671,
    "msg": "pubsub processloop shutting down",
    "name": "127.0.0.1:0",
    "libp2p": "pubsub",
    "time": "2024-04-17T19:43:36.354601-04:00"
}

Test Plan

Ran scenarios, ensured progress
Deployed networks, collected/analyzed regular performance metrics - 4-6k TPS on scenario1s-p2p

p2p net scenario	TPS	MEM
without pubsub
with pubsub (default)

codecov · 2024-03-18T21:07:20Z

Codecov Report

Attention: Patch coverage is 53.52113% with 66 lines in your changes are missing coverage. Please review.

Project coverage is 56.02%. Comparing base (411837f) to head (f1f03a7).

Files	Patch %	Lines
network/p2p/logger.go	0.00%	41 Missing ⚠️
network/p2p/mux.go	78.12%	11 Missing and 3 partials ⚠️
network/p2pNetwork.go	69.56%	7 Missing ⚠️
logging/log.go	0.00%	2 Missing ⚠️
network/p2p/http.go	0.00%	1 Missing ⚠️
network/p2p/streams.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##           feature/p2p    #5962    +/-   ##
=============================================
  Coverage        56.01%   56.02%            
=============================================
  Files              495      497     +2     
  Lines            69009    69130   +121     
=============================================
+ Hits             38657    38731    +74     
- Misses           27730    27774    +44     
- Partials          2622     2625     +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

algorandskiy · 2024-05-10T20:33:35Z

test/heapwatch/topology-extract-p2p.py

@@ -0,0 +1,66 @@
+"""


the idea is to have it dependency-free so it can be run on a remote machine with logs

codecov-commenter · 2024-05-23T19:57:09Z

Codecov Report

Attention: Patch coverage is 31.73077% with 71 lines in your changes missing coverage. Please review.

Project coverage is 56.12%. Comparing base (8f9ad90) to head (e59fd47).

Files	Patch %	Lines
network/p2p/logger.go	0.00%	41 Missing ⚠️
network/p2pNetwork.go	63.15%	10 Missing and 4 partials ⚠️
network/p2p/p2p.go	0.00%	6 Missing ⚠️
network/p2p/streams.go	0.00%	4 Missing ⚠️
config/localTemplate.go	0.00%	3 Missing ⚠️
logging/log.go	0.00%	2 Missing ⚠️
network/p2p/http.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           feature/p2p    #5962      +/-   ##
===============================================
- Coverage        56.13%   56.12%   -0.02%     
===============================================
  Files              487      488       +1     
  Lines            69327    69396      +69     
===============================================
+ Hits             38919    38946      +27     
- Misses           27760    27800      +40     
- Partials          2648     2650       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This reverts commit 952b398.

This reverts commit fe0d4bb.

cce · 2024-06-12T18:51:29Z

network/p2p/logger.go

+			loggingFields[f.Key] = f.Interface
+		} else {
+			loggingFields[f.Key] = f.Integer
+		}


I guess it is OK to ignore the field type because they all boil down to one of these three types?

network/p2p/p2p.go

network/p2pNetwork.go

cce · 2024-06-12T19:28:55Z

network/p2pNetwork.go

-	n.log.With("event", event).With("remote", addr).With("local", localAddr).Infof(msg, p2ppeer.String())
+	localAddr, has := n.Address()
+	if !has {
+		n.log.Warn("Could not get local address")


How often does this happen?

not sure but it did happen few times - discovered when parsed node.log files for topology images.

algorandskiy added Enhancement p2p Work related to the p2p project labels Mar 18, 2024

algorandskiy self-assigned this Mar 18, 2024

algorandskiy force-pushed the pavel/p2p-scenarios branch 4 times, most recently from 69cb867 to a5ccfcb Compare April 2, 2024 20:15

algorandskiy requested a review from cce April 17, 2024 23:29

algorandskiy force-pushed the pavel/p2p-scenarios branch 2 times, most recently from 3a7e767 to 79251d0 Compare April 19, 2024 21:31

algorandskiy changed the title ~~WIP: p2p: test scenarios support~~ p2p: test scenarios support Apr 20, 2024

algorandskiy marked this pull request as ready for review April 20, 2024 00:12

algorandskiy force-pushed the pavel/p2p-scenarios branch from 09a16c1 to d2d2e3a Compare May 10, 2024 20:30

algorandskiy commented May 10, 2024

View reviewed changes

algorandskiy force-pushed the pavel/p2p-scenarios branch 2 times, most recently from 92b7d16 to 9c63601 Compare May 15, 2024 19:30

algorandskiy added 12 commits June 7, 2024 15:32

add p2p dnsadddr support to netdeploy

356dd08

add a minimal p2p cluster test scenario

b86c22f

Add logging

3110bf7

fixes: peerID string, network data dir vs genesis dir

c01e430

fix tests

7de8e80

Make scenarion1s rounds smaller

4f13174

add scenario1s-p2p

06d52dd

rename hw-p2p to hw-tiny-p2p

3255688

add hw-small-p2p

43f91c1

enable metrics on all hello-world-small-p2p nodes

e2d6d88

heapwatch scripts fix - metric and empty {}

0b1198a

emable runtime mertics in hello-world-small-p2p

d842a15

algorandskiy added 17 commits June 7, 2024 15:35

copy libp2p muxer to use forked yamux

952b398

heapwatch metrics viz: fix ip to names resolution

8f6e578

metric_viz: store html

0cf4dc4

metric_viz: nodes filtering

f0ed6e5

metric_viz fixes

739f10f

metrics_viz: support metrics with tags

06a0282

fix opensensus metrics names

ea3d65c

metrics_viz refactoring, add metrics_aggs

73b9519

topology scripts: support missing local addr

9c3d134

use connection direction for conn counting instead of stream direction

2666244

agreement node.log aggregator

1358067

Add connection timestamp to topology parser

6f2d5fc

add DNSSecurityFlags=dnssecTXT flag to config

c0d2e78

fix outgoing conn counting

e794266

add exp backoff to mesh thread

d05aa9c

post rebase fixes

b27c949

linter fixes

cb06762

algorandskiy force-pushed the pavel/p2p-scenarios branch from e105157 to cb06762 Compare June 7, 2024 19:44

algorandskiy added 3 commits June 7, 2024 15:45

round time in agreement log

5942217

Revert "copy libp2p muxer to use forked yamux"

78a35fe

This reverts commit 952b398.

Revert "wip: try yamux high prio fork"

e59fd47

This reverts commit fe0d4bb.

algorandskiy requested a review from gmalouf June 11, 2024 18:26

cce reviewed Jun 12, 2024

View reviewed changes

network/p2p/p2p.go Show resolved Hide resolved

cce reviewed Jun 12, 2024

View reviewed changes

network/p2pNetwork.go Show resolved Hide resolved

cce reviewed Jun 12, 2024

View reviewed changes

cce approved these changes Jun 12, 2024

View reviewed changes

algorandskiy merged commit 052792d into algorand:feature/p2p Jun 12, 2024
16 checks passed

algorandskiy mentioned this pull request Jun 13, 2024

network: p2p traffic exchange for algorand node #5939

Merged

6 tasks

algorandskiy mentioned this pull request Jul 23, 2024

Network Testing: Come up with new/hybrid topologies and modified scenarios #6076

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

p2p: test scenarios support #5962

p2p: test scenarios support #5962

algorandskiy commented Mar 18, 2024 •

edited

Loading

codecov bot commented Mar 18, 2024 •

edited

Loading

algorandskiy May 10, 2024

codecov-commenter commented May 23, 2024 •

edited by codecov bot

Loading

cce Jun 12, 2024

cce Jun 12, 2024

algorandskiy Jun 12, 2024

p2p: test scenarios support #5962

p2p: test scenarios support #5962

Conversation

algorandskiy commented Mar 18, 2024 • edited Loading

Summary

Test Plan

codecov bot commented Mar 18, 2024 • edited Loading

Codecov Report

algorandskiy May 10, 2024

Choose a reason for hiding this comment

codecov-commenter commented May 23, 2024 • edited by codecov bot Loading

Codecov Report

cce Jun 12, 2024

Choose a reason for hiding this comment

cce Jun 12, 2024

Choose a reason for hiding this comment

algorandskiy Jun 12, 2024

Choose a reason for hiding this comment

algorandskiy commented Mar 18, 2024 •

edited

Loading

codecov bot commented Mar 18, 2024 •

edited

Loading

codecov-commenter commented May 23, 2024 •

edited by codecov bot

Loading