feat(telemetry): send telemetry messages when GRANDPA receieves commit or vote messages #2015

kishansagathiya · 2021-11-11T10:36:05Z

Changes

Send afg.received_commit when grandpa receives a commit message.
Send afg.received_precommit or afg.received_prevote when grandpa
receives a vote message

Tests

Setup a local telemetry server using substrate-telemetry.

Use the branch print-payload-type from my fork https://github.com/kishansagathiya/substrate-telemetry (It print the message received, type of message and error. From what I have played around upstream doesn't do that.)

Run telementry backend (telemetry_core and telemetry_shard).

Run gossamer by setting telemetry url to the local server.

In order to run perform grandpa rounds, we will need to run a dev net. https://hackmd.io/@nZ-twauPRISEa6G9zg3XRw/B1RFO3uDt shows how to do that.
I ran it with 3 nodes. After running all the nodes, you should be able to see in telemetry_shard logs that we are receiving all 3 expected telemetry messages from time to time.

Issues

Primary Reviewer

@noot

kishansagathiya · 2021-11-11T10:36:31Z

Waiting on #1966 to get merged

codecov · 2021-11-11T11:10:28Z

Codecov Report

Merging #2015 (77a8a09) into development (c17b53a) will increase coverage by 0.14%.
The diff coverage is 85.29%.

@@               Coverage Diff               @@
##           development    #2015      +/-   ##
===============================================
+ Coverage        60.48%   60.62%   +0.14%     
===============================================
  Files              201      202       +1     
  Lines            27216    27282      +66     
===============================================
+ Hits             16461    16540      +79     
+ Misses            8856     8842      -14     
- Partials          1899     1900       +1

Flag	Coverage Δ
unit-tests	`60.62% <85.29%> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dot/telemetry/telemetry.go	`70.23% <ø> (ø)`
dot/telemetry/txpool_import.go	`100.00% <ø> (ø)`
lib/grandpa/grandpa.go	`58.48% <66.66%> (+0.09%)`	⬆️
lib/grandpa/vote_message.go	`82.78% <76.00%> (-0.78%)`	⬇️
lib/grandpa/message_handler.go	`64.42% <81.25%> (+0.60%)`	⬆️
dot/telemetry/afg_received.go	`100.00% <100.00%> (ø)`
dot/network/light.go	`86.05% <0.00%> (+0.79%)`	⬆️
dot/network/connmgr.go	`75.78% <0.00%> (+1.05%)`	⬆️
lib/blocktree/blocktree.go	`54.90% <0.00%> (+1.09%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c17b53a...77a8a09. Read the comment docs.

danforbes · 2021-11-13T14:13:57Z

Is this ready for review?

Send `afg.received_commit` when grandpa receives a commit message. Send `afg.received_precommit` or `afg.received_prevote` when grandpa receives a vote message Closes #1840 Closes #1839 Closes #1838

kishansagathiya · 2021-11-14T20:53:52Z

Is this ready for review?

It wasn't at the time, but it is ready now.

lib/grandpa/vote_message.go

noot

looks good!

qdm12

Nice 👍
Just a few comments here and there

dot/telemetry/afg_received.go

dot/telemetry/telemetry.go

dot/telemetry/telemetry_test.go

lib/grandpa/grandpa.go

lib/grandpa/message_handler.go

lib/grandpa/vote_message.go

dot/telemetry/afg_received.go

# [0.6.0](v0.5.0...v0.6.0) (2021-12-03) ### Bug Fixes * **babe:** Fix extrinsic format in block. ([#1530](#1530)) ([1a03b2a](1a03b2a)) * **ci:** add missing go-version matrix to fix development branch CI ([#2037](#2037)) ([6babe76](6babe76)) * **cmd/cfg:** Use Babe Lead value from toml config ([#2032](#2032)) ([06aa3e3](06aa3e3)) * cmd/gossamer: Generate random name if --name flag not set ([#1506](#1506)) ([3c05a88](3c05a88)) * confirm block import notifier is closed properly ([#1736](#1736)) ([ad2d85e](ad2d85e)) * **docs:** improve build-spec usage docs ([#1706](#1706)) ([2e164b4](2e164b4)) * **dot/core:** Add only extrinsic during chain reorg. ([#1609](#1609)) ([29413d4](29413d4)) * **dot/core:** Batch process transaction message. ([#1780](#1780)) ([0064836](0064836)) * **dot/core:** check transaction Validity.Propagate field to determine whether to propagate tx ([#1643](#1643)) ([81f23cc](81f23cc)) * **dot/core:** Fix handle transaction message test. ([#1607](#1607)) ([58b8725](58b8725)) * **dot/network, lib/grandpa:** fix handshake decoding and grandpa message handler sigabort ([#1631](#1631)) ([887f72c](887f72c)) * **dot/network, lib/grandpa:** fix node sync, improve devnet finality ([bcc7935](bcc7935)) * **dot/network:** add nil checks in connManager ([#2069](#2069)) ([7f9c042](7f9c042)) * **dot/network:** Check for size when decoding leb128. ([#1634](#1634)) ([d082b9e](d082b9e)) * **dot/network:** check if peer supports protocol ([#1617](#1617)) ([6bf66a4](6bf66a4)) * **dot/network:** decrease DHT find peers interval for gssmr nodes ([#1703](#1703)) ([08516a0](08516a0)) * **dot/network:** fix bugs in notifications protocol handlers; add metrics for inbound/outbound streams ([#2010](#2010)) ([8c2993d](8c2993d)) * **dot/network:** fix dht connection on discovery on devnet ([#2059](#2059)) ([da065b8](da065b8)) * **dot/network:** fix discovery between gossamer nodes ([#1594](#1594)) ([f4c79d3](f4c79d3)) * **dot/network:** fix justification request at head logic ([#1510](#1510)) ([98d1413](98d1413)) * **dot/network:** fix memory allocations with `sizedBufferPool` ([#1963](#1963)) ([e0b126b](e0b126b)) * **dot/network:** Fix missing digest in header ([#2092](#2092)) ([21ea85e](21ea85e)) * **dot/network:** Fix notification handshake and reuse stream. ([#1545](#1545)) ([a632dc4](a632dc4)) * **dot/network:** fix receiving notifications messages from substrate peers ([#1517](#1517)) ([fdf3c53](fdf3c53)) * **dot/network:** fix stream manager tests ([#1683](#1683)) ([e02eca4](e02eca4)) * **dot/network:** implement a handshake timeout ([#1615](#1615)) ([87c2f63](87c2f63)) * **dot/network:** Implement time based handle transaction ([#1942](#1942)) ([dd08424](dd08424)) * **dot/network:** move low reputation peer removal from network ConnManager to peer scoring logic (dot/peerstate) ([#2068](#2068)) ([ac16285](ac16285)), closes [#2039](#2039) * **dot/network:** Return on EOF error while reading stream. ([#1733](#1733)) ([f447eac](f447eac)) * **dot/network:** split stored streams and handshakeData into inbound and outbound ([#1553](#1553)) ([637050b](637050b)) * **dot/network:** update notificationsProtocol handshakeData to sync.Map ([#1492](#1492)) ([22f7269](22f7269)) * **dot/node:** Start websocket server only with `--ws` flag ([#1671](#1671)) ([6ecef3b](6ecef3b)) * **dot/state, lib/babe, lib/trie:** improve syncing between gossamer authority nodes ([#1613](#1613)) ([ca99fbf](ca99fbf)) * **dot/state, lib/grandpa:** update justification and SignedVote handling in database ([#1682](#1682)) ([bbdcd6f](bbdcd6f)) * **dot/state,dot/network:** improve memory usage when syncing ([#1491](#1491)) ([3b2ad8d](3b2ad8d)) * **dot/state:** add StorageState Lock/Unlock API for usage by babe and sync ([#1700](#1700)) ([3c22ace](3c22ace)) * **dot/state:** fix deadlock, fixes bootstrap syncing ([#1959](#1959)) ([dd80c09](dd80c09)) * **dot/state:** fix usage of trie.Snapshot ([#1489](#1489)) ([3880a40](3880a40)) * **dot/state:** track runtime per-block, fix runtime upgrades differing between forks ([#1638](#1638)) ([e133884](e133884)) * **dot/state:** update `*state.BlockState.AddBlockToBlockTree` to store block in `unfinalisedBlocksMap` ([#2006](#2006)) ([55d997f](55d997f)) * **dot/sync:** add nil header checks ([#2099](#2099)) ([a7d4be0](a7d4be0)) * **dot/sync:** fix block request and response logic ([#1907](#1907)) ([9c6283e](9c6283e)) * **dot/sync:** fix creating block response, fixes node sync between gossamer nodes ([#1572](#1572)) ([1328c80](1328c80)) * **dot/telemetry:** refactor telemetry to reduce CPU usage ([#1597](#1597)) ([bc31ac7](bc31ac7)) * **dot/types:** *types.Body to be of type []types.Extrinsic ([#1807](#1807)) ([4c09715](4c09715)) * **dot/types:** fix max value for digest ([#1687](#1687)) ([48405e7](48405e7)) * **dot:** fix `TestNewNode` ([#2070](#2070)) ([42908d0](42908d0)) * fix edit link ([#1507](#1507)) ([5089327](5089327)) * fix Kusama sync; add storageState lock in core.HandleTransactionMessage ([#1783](#1783)) ([1d688e4](1d688e4)) * **lib/babe, lib/runtime/wasmer:** fixes for v0.9.8+ runtime ([#2075](#2075)) ([2f9f80c](2f9f80c)) * **lib/babe:** add `--babe-lead` flag, update epoch handling logic ([#1895](#1895)) ([7abcce6](7abcce6)) * **lib/babe:** add pre-runtime digest before calling initialize_block ([#1581](#1581)) ([c1b26d3](c1b26d3)) * **lib/babe:** always use 2/3 of slot to produce block, re-add potentially valid txs to queue ([#1679](#1679)) ([cf93ad3](cf93ad3)) * **lib/babe:** call AddBlock in BABE synchronously ([#1585](#1585)) ([86acc43](86acc43)) * **lib/babe:** fix BABE state storing after building block ([#1536](#1536)) ([1a3dea2](1a3dea2)) * **lib/babe:** fix err log ([#1801](#1801)) ([a96f06a](a96f06a)) * **lib/babe:** fix setting first slot of network, fix loading BABE epoch params ([#1640](#1640)) ([5c3dbfe](5c3dbfe)) * **lib/babe:** fix timing for transition between epochs ([#1636](#1636)) ([57027db](57027db)) * **lib/blocktree:** fix blocktree bug ([#2060](#2060)) ([c17b53a](c17b53a)) * **lib/blocktree:** fix potential nil pointer dereference in `HighestCommonAncestor`, core `handleBlocksAsync` ([#1993](#1993)) ([f7f4463](f7f4463)) * **lib/blocktree:** fix setting leaves after blocktree pruning ([#1605](#1605)) ([58c0854](58c0854)) * **lib/blocktree:** removes the inconsistency to choose a deepest leaf ([#2094](#2094)) ([43d68e3](43d68e3)) * **lib/crypto/ed25519:** update ed25519 to use go-schnorrkel bip39 derivation ([#1488](#1488)) ([dfb95d2](dfb95d2)) * **lib/genesis:** Update missing and incorrect fields in genesis file. ([#1681](#1681)) ([8207704](8207704)) * **lib/grandpa:** fix grandpa stall and various bugs ([#1708](#1708)) ([67c93f4](67c93f4)) * **lib/grandpa:** fix grandpa vote message switch ([#2095](#2095)) ([461890c](461890c)) * **lib/grandpa:** fix threshold checking to be strictly greater than 2/3 ([#1891](#1891)) ([66ffe51](66ffe51)) * **lib/grandpa:** use `defaultGrandpaInterval` if not set, fixes error on startup ([#1982](#1982)) ([75627b5](75627b5)) * **lib/runtime/life:** remove import C from life ([#1923](#1923)) ([ed507d2](ed507d2)) * **lib/runtime:** update HOST_API_TEST_RUNTIME_URL to reference specific commit ([#1885](#1885)) ([666ed06](666ed06)) * **log-levels:** do not ignore configuration file log levels ([#2016](#2016)) ([80879b2](80879b2)) * pending bubble hidden after block included ([#1592](#1592)) ([5826322](5826322)) * persist node name ([#1543](#1543)) ([88b88f2](88b88f2)) * **pprof:** only run pprof service if enabled ([#2073](#2073)) ([55669c5](55669c5)) * **release:** Trigger release when pushed to main branch. ([#1566](#1566)) ([d445c97](d445c97)) * **rpc/subscription:** subscribe runtime version notify when version changes ([#1686](#1686)) ([9a76d39](9a76d39)) * Staging CI workflow ([#2034](#2034)) ([84ec792](84ec792)) * **trie:** memory leak fix in `lib/trie` ([#2009](#2009)) ([0ad5eb7](0ad5eb7)) * update deprecated package ([#1603](#1603)) ([f195204](f195204)) * update go-schnorrkel version ([#1557](#1557)) ([b86c7ff](b86c7ff)) * update gssmr genesis to use pallet_babe::SameAuthoritiesForever ([#1696](#1696)) ([fb0a751](fb0a751)) * update HOST_API_TEST_RUNTIME_URL ([#1898](#1898)) ([2ef59a8](2ef59a8)) * **utils:** create a specific folder for database ([#1598](#1598)) ([8c67795](8c67795)) ### Features * add --chain dev option ([#1561](#1561)) ([04a2969](04a2969)) * Add properties and chainId on build-spec command ([#1520](#1520)) ([b18290c](b18290c)) * cmd: implement import-runtime subcommand ([#1483](#1483)) ([d82b2da](d82b2da)) * **cmd/gossamer:** implement --telemetry-url parameter ([#1890](#1890)) ([b202e89](b202e89)), closes [#1502](#1502) * **cmd:** implement offline pruning of state trie ([#1564](#1564)) ([af9c925](af9c925)) * **devnet:** Local Gossamer Devnet ([#2008](#2008)) ([a520001](a520001)) * **dot/network, lib/grandpa:** request justification on receiving NeighbourMessage, verify justification on receipt ([#1529](#1529)) ([e1f9f42](e1f9f42)) * **dot/network:** Add cache for network message. ([#1511](#1511)) ([accaf69](accaf69)) * **dot/network:** add propagate return bool to messageHandler func type to determine whether to propagate message or not ([#1555](#1555)) ([0d6f488](0d6f488)) * **dot/network:** implement persistent peers functionality ([#1512](#1512)) ([7850532](7850532)) * **dot/network:** implement streamManager to cleanup not recently used streams ([#1611](#1611)) ([ba861bf](ba861bf)) * **dot/network:** request block justifications when near head ([#1499](#1499)) ([ae7012b](ae7012b)) * **dot/peerset:** Implement peer scoring ([#1791](#1791)) ([1c989ad](1c989ad)) * **dot/rpc/modules:** add `system_addReservedPeer` and `system_removeReservedPeer` RPC call ([#1712](#1712)) ([dba5922](dba5922)) * **dot/rpc:** Add `system_localListenAddresses` RPC call ([#1689](#1689)) ([c981d2e](c981d2e)) * **dot/rpc:** Implement `childstate_getKeys` rpc call ([#1800](#1800)) ([9b2f41e](9b2f41e)) * **dot/rpc:** implement sync_state_genSyncSpec RPC call ([#1827](#1827)) ([2186caf](2186caf)) * **dot/state:** implement online pruning of historical state tries ([#1596](#1596)) ([3eb9399](3eb9399)) * **dot/sync:** implement codeSubstitutes ([#1635](#1635)) ([d87aaeb](d87aaeb)) * dot/telemetry: Implement basic telemetry connection ([#1497](#1497)) ([fcb4159](fcb4159)) * **dot/telemetry:** Added connection retry ([#1904](#1904)) ([579a791](579a791)) * **dot/telemetry:** Added more telemetry messages in grandpa client ([#2043](#2043)) ([2e57d15](2e57d15)), closes [#1841](#1841) [#1842](#1842) * **dot/telemetry:** implement notify.finalized telemetry interface ([#1877](#1877)) ([de1a60d](de1a60d)) * **dot/telemetry:** implement substrate_number_leaves metrics ([#1926](#1926)) ([69823c0](69823c0)) * **dot/telemetry:** implement telemetry message network_state ([#1618](#1618)) ([a81844e](a81844e)) * **flags:** read log levels from flags ([#1953](#1953)) ([9694e46](9694e46)) * implement ext_default_child_storage_storage_kill_version_2 ([#1799](#1799)) ([c2908ae](c2908ae)) * implement ext_offchain_index_set_version_1 for wasmer runtime ([#1739](#1739)) ([96c30a6](96c30a6)) * **lib/babe:** add check of types.ConfigData.SecondarySlots for disabling secondary verification ([#1910](#1910)) ([cd27ae4](cd27ae4)) * **lib/grandpa:** fully verify justifications using GrandpaState ([#1544](#1544)) ([028d25e](028d25e)) * **lib/grandpa:** Include equivocatory nodes while creating justification ([#1911](#1911)) ([aca86b6](aca86b6)) * **lib/grandpa:** send NeighbourMessage to peers ([#1558](#1558)) ([322ccf9](322ccf9)) * **lib/runtime/wasmer:** implement ext_default_child_storage_storage_kill_version_3 ([#1878](#1878)) ([a719a60](a719a60)) * **lib/runtime/wasmer:** implement ext_offchain_local_storage_version_1 ([#1821](#1821)) ([0f63b17](0f63b17)) * **lib/runtime:** Implement `ext_offchain_http_request_add_header_version_1` host function ([#1994](#1994)) ([0a30b3d](0a30b3d)) * **lib/runtime:** Implement `ext_offchain_http_request_start_version_1` host function ([#1947](#1947)) ([974b1fc](974b1fc)) * **lib/runtime:** Implement `trie_blake2_256_verify_proof` host function ([#1920](#1920)) ([506565d](506565d)) * **lib/trie:** Implement `verify_proof` function ([#1883](#1883)) ([67bb5ef](67bb5ef)) * **lib/trie:** Implement limit for trie.ClearPrefix ([#1905](#1905)) ([becec9e](becec9e)) * **lib/trie:** Parallel hash trie. ([#1657](#1657)) ([22827e7](22827e7)) * **pprof:** Pprof HTTP server service ([#1991](#1991)) ([ce24ea9](ce24ea9)) * **rpc/subscription:** implement state_unsubscribeStorage ([#1574](#1574)) ([7574f10](7574f10)) * **rpc:** Implement `childstate_getChildStorage` RPC call ([#1832](#1832)) ([3d949f2](3d949f2)) * **rpc:** Implement `childstate_getStorageHash` RPC call ([#1805](#1805)) ([e539bd3](e539bd3)) * **rpc:** Implement `childstate_getStorageSize` RPC call ([#1810](#1810)) ([a04deb6](a04deb6)) * **rpc:** Implement `payment_queryInfo` RPC call ([#1826](#1826)) ([7a5deec](7a5deec)) * **rpc:** Implement `state_getReadProof` rpc call ([#1768](#1768)) ([865f80f](865f80f)) * **runtime:** implement custom logging handler that print function name ([#1825](#1825)) ([2b1276d](2b1276d)) * **telemetry:** send telemetry messages when GRANDPA receieves commit or vote messages ([#2015](#2015)) ([7bf40e1](7bf40e1)), closes [#1840](#1840) [#1839](#1839) [#1838](#1838) * **telemetry:** send txpool.import telemetry msg ([#1966](#1966)) ([ffc81bf](ffc81bf)) ### Reverts * Revert "feat(dot/rpc) implement `author_hasSessionKeys` RPC call (#1704)" (#1714) ([65380fd](65380fd)), closes [#1704](#1704) [#1714](#1714)

github-actions · 2021-12-03T21:03:26Z

🎉 This PR is included in version 0.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

…t or vote messages (ChainSafe#2015) * telemetry when GRANDPA receieve commit or vote messages Send `afg.received_commit` when grandpa receives a commit message. Send `afg.received_precommit` or `afg.received_prevote` when grandpa receives a vote message Closes ChainSafe#1840 Closes ChainSafe#1839 Closes ChainSafe#1838

kishansagathiya changed the title ~~telemetry when GRANDPA receieve commit or vote messages~~ feat(telemetry): send telemetry messages when GRANDPA receieve commit or vote messages Nov 11, 2021

kishansagathiya changed the title ~~feat(telemetry): send telemetry messages when GRANDPA receieve commit or vote messages~~ feat(telemetry): send telemetry messages when GRANDPA receieves commit or vote messages Nov 11, 2021

kishansagathiya added 2 commits November 15, 2021 02:10

telemetry when GRANDPA receieve commit or vote messages

fd6f328

Send `afg.received_commit` when grandpa receives a commit message. Send `afg.received_precommit` or `afg.received_prevote` when grandpa receives a vote message Closes #1840 Closes #1839 Closes #1838

Merge branch 'development' into kishan/telemetry/afg-received

6da8b5d

kishansagathiya force-pushed the kishan/telemetry/afg-received branch from 3fb77d6 to 6da8b5d Compare November 14, 2021 20:52

kishansagathiya marked this pull request as ready for review November 14, 2021 20:54

kishansagathiya requested review from arijitAD, EclesioMeloJunior, edwardmack, jimjbrettj, noot, omar391 and timwu20 as code owners November 14, 2021 20:54

kishansagathiya added 2 commits November 15, 2021 12:01

fix tests

897fd37

one more small fix

cd3382f

EclesioMeloJunior requested changes Nov 15, 2021

View reviewed changes

lib/grandpa/vote_message.go Show resolved Hide resolved

noot approved these changes Nov 16, 2021

View reviewed changes

addressed comments

e484db7

kishansagathiya requested a review from EclesioMeloJunior November 17, 2021 08:07

kishansagathiya added 2 commits November 22, 2021 13:40

Merge branch 'development' into kishan/telemetry/afg-received

48053e9

Merge branch 'development' into kishan/telemetry/afg-received

f9e7b52

qdm12 suggested changes Nov 26, 2021

View reviewed changes

kishansagathiya added 2 commits November 29, 2021 19:17

addressed some reviews

21cb548

Merge branch 'development' into kishan/telemetry/afg-received

487fd15

kishansagathiya requested a review from qdm12 November 29, 2021 14:42

EclesioMeloJunior approved these changes Nov 29, 2021

View reviewed changes

qdm12 approved these changes Nov 29, 2021

View reviewed changes

kishansagathiya added 2 commits November 29, 2021 22:51

Merge branch 'development' into kishan/telemetry/afg-received

e40e281

Merge branch 'development' into kishan/telemetry/afg-received

77a8a09

kishansagathiya merged commit 7bf40e1 into development Nov 30, 2021

kishansagathiya deleted the kishan/telemetry/afg-received branch November 30, 2021 07:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(telemetry): send telemetry messages when GRANDPA receieves commit or vote messages #2015

feat(telemetry): send telemetry messages when GRANDPA receieves commit or vote messages #2015

kishansagathiya commented Nov 11, 2021 •

edited

Loading

kishansagathiya commented Nov 11, 2021

codecov bot commented Nov 11, 2021 •

edited

Loading

danforbes commented Nov 13, 2021

kishansagathiya commented Nov 14, 2021

noot left a comment

qdm12 left a comment

github-actions bot commented Dec 3, 2021

feat(telemetry): send telemetry messages when GRANDPA receieves commit or vote messages #2015

feat(telemetry): send telemetry messages when GRANDPA receieves commit or vote messages #2015

Conversation

kishansagathiya commented Nov 11, 2021 • edited Loading

Changes

Tests

Issues

Primary Reviewer

kishansagathiya commented Nov 11, 2021

codecov bot commented Nov 11, 2021 • edited Loading

Codecov Report

danforbes commented Nov 13, 2021

kishansagathiya commented Nov 14, 2021

noot left a comment

Choose a reason for hiding this comment

qdm12 left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 3, 2021

kishansagathiya commented Nov 11, 2021 •

edited

Loading

codecov bot commented Nov 11, 2021 •

edited

Loading