conduit: Initial conduit pipeline tool. #1326

winder · 2022-11-10T23:09:51Z

Summary

Add new conduit data pipeline command. The data processing component of Indexer has been removed and turned into a modular data pipeline. Data loading can be configured and extended by enabling different plugins.

Plugins are separated into three categories:

importer: data source for block information.
processor: manipulate, filter or enrich data.
exporter: do something with the data, like send it to a database.

Indexer has been modified to use a pre-configured conduit pipeline internally.

Some notable features that are available through conduit:

Transaction filtering - filter out transactions that you do not want.
Data pruning - automatically discard old transactions from the postgres database.

For more information, see the documentation.

Test Plan

New unit tests.
Production network catchup and data validation tests.

* Local Ledger (#1011) * integrate block processor * Local Ledger Deployment (#1013) * add simple local ledger migration * add deleted opts * fast catchup (#1023) * add fast catchup * Localledger merge (#1036) * return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> * recovery scenario (#1024) * handle ledger recovery scenario * refactor create genesis block (#1026) * refactor create genesis block * Adds Local Ledger Readme (#1035) * Adds Local Ledger Readme Resolves #4109 Starts Readme docs * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Removed troubleshooting section Co-authored-by: Will Winder <[email protected]> * update ledger file path and migration (#1042) * LocalLedger Refactoring + Catchpoint Service (#1049) Part 1 cleanup genesis file access. put node catchup into a function that can be swapped out with the catchup service. pass the indexer logger into the block processor. move open ledger into a util function, and move the initial state util function into a new ledger util file. add initial catchupservice implementation. move ledger init from daemon.go to constructor. Merge multiple read genesis functions. Part 2 Merge local_ledger migration package into blockprocessor. Rename Migration to Initialize Use logger in catchup service catchup Part 3 Update submodule and use NewWrappedLogger. Make util.CreateInitState private * build: merge develop into localledger/integration (#1062) * Ledger init status (#1058) * Generate an error if the catchpoint is not valid for initialization. (#1075) * Use main logger in handler and fetcher. (#1077) * Switch from fullNode catchup to catchpoint catchup service. (#1076) * Refactor daemon, add more tests (#1039) Refactors daemon cmd into separate, testable pieces. * Merge develop into localledger/integration (#1083) * Misc Local Ledger cleanup (#1086) * Update processor/blockprocessor/initialize.go Co-authored-by: Zeph Grunschlag <[email protected]> * commit * fix function call args * RFC-0001: Rfc 0001 impl (#1069) Adds an Exporter interface and a noop exporter implementation with factory methods for construction * Fix test errors * Add/fix tests * Add postgresql_exporter tests * Update config loading * Change BlockExportData to pointers * Move and rename ExportData * Add Empty func to BlockData * Add comment Co-authored-by: shiqizng <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Updates for changes to our plugin design based on feedback in RFCs * Merges in submodule update for test failure fixes

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Update READEME * Add initial Conduit.md * Add example exporter, exporter README.md

* importer interface | algod importer implementation * added new tests | modified importer interface * exported mockAlgodServer from test * updated algodImporter plugin * Update importers/algod/algod_importer.go Co-authored-by: Eric Warehime <[email protected]> Co-authored-by: Eric Warehime <[email protected]>

* Added Processor Added processor functionality * pr updates * Pr * PR comments * Reduce proto checks * Linting * Pr * PR comments * PR comments

* Use config to control noop exporter round start

* Initial files * Initial Conduit Binary Resolves #1165 Creates an initial framework for conduit binary production. Introduces a data directory flag as well as init and shutdown (basic) functionality. * linting * PR comments * init via importer * Passing test * e2e test passing * Modify makefile so that algorand-indexer is built * Pr changes * Log level changes * Modify bindflagset for conduit * Makefile changes * Makefile * PR updates

* Add waits/retries to algod importer

* conduit pipeline run loop implementation

* Adds unit tests Resolves #1192 Adds unit tests to the config directory * Extra assert * PR comments * Wait for round increase * Added tests * context fix

Merge indexer/develop

* Adds cpu profile and pid flag Resolves #1172 Adds the options for cpu profiling and pid flags as well as unit tests * Update conduit/pipeline.go Co-authored-by: shiqizng <[email protected]> * linting * Added pid file function * rebase Co-authored-by: shiqizng <[email protected]>

* Convert indexer daemon to conduit

* adding file exporter

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <[email protected]> Co-authored-by: Will Winder <[email protected]> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: algolucky <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: John Lee <[email protected]> Co-authored-by: Or Aharonee <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: algolucky <[email protected]>

* Initial Filter Processor Resolves #1100 Adds dynamic filter processor and initial tests * Nits * Update processors/filterprocessor/filter_processor.go Co-authored-by: Eric Warehime <[email protected]> * Update processors/filterprocessor/filter_processor_test.go Co-authored-by: Eric Warehime <[email protected]> * Update processors/filterprocessor/filter_processor_test.go Co-authored-by: Eric Warehime <[email protected]> * PR comments * Pr comments * PR updates * PR updates * Pr updates * PR updates * PR updates * PR updates * Changed some to any and const to exact * Test failure fix * Added more unit tests * Config comments Co-authored-by: Eric Warehime <[email protected]>

* Add metrics reporting to conduit pipeline

* Changed tag format * Updated generation code * Added filtering of numerical values * PR comments * imports

* Add panic recovers Resolves #263 Adds recovery functions in main loop of conduit as well as in a go-routine in the start function of the pipeline * helper function

Resolves #1245 Explicitly spell out empty data directory as well as condensed error messages when running conduit

Resolves #1283 Adds none operation to fields

Co-authored-by: Will Winder <[email protected]>

codecov · 2022-11-14T16:17:30Z

Codecov Report

Merging #1326 (9f32987) into develop (39159bd) will decrease coverage by 0.40%.
The diff coverage is 56.04%.

@@             Coverage Diff             @@
##           develop    #1326      +/-   ##
===========================================
- Coverage    61.11%   60.71%   -0.41%     
===========================================
  Files           52       76      +24     
  Lines         8512    10909    +2397     
===========================================
+ Hits          5202     6623    +1421     
- Misses        2848     3699     +851     
- Partials       462      587     +125

Impacted Files	Coverage Δ
api/error_messages.go	`100.00% <ø> (ø)`
api/generated/common/routes.go	`46.51% <ø> (ø)`
conduit/errors.go	`0.00% <0.00%> (ø)`
conduit/gen/lookup.go	`0.00% <0.00%> (ø)`
...ocessors/blockprocessor/internal/catchupservice.go	`69.66% <ø> (ø)`
...plugins/processors/filterprocessor/gen/generate.go	`0.00% <0.00%> (ø)`
idb/idb.go	`69.35% <ø> (ø)`
idb/postgres/internal/encoding/types.go	`66.66% <ø> (ø)`
idb/postgres/internal/writer/util.go	`100.00% <ø> (ø)`
idb/sig_type.go	`64.00% <0.00%> (ø)`
... and 52 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

conduit/plugins/exporters/noop/noop_exporter.go

conduit/plugins/exporters/example/example_exporter.go

…a1099c` (#1304)

Eric-Warehime and others added 30 commits July 21, 2022 08:18

Add all PluginTypes (#1136)

e2c50a0

Conduit: Rfc updates (#1139)

2c945b4

* Updates for changes to our plugin design based on feedback in RFCs * Merges in submodule update for test failure fixes

docs: Plugin docs (#1150)

f2b422f

* Update READEME * Add initial Conduit.md * Add example exporter, exporter README.md

Create RFC for processor (#1140)

3d2720c

* Added Processor Added processor functionality * pr updates * Pr * PR comments * Reduce proto checks * Linting * Pr * PR comments * PR comments

Use config to control noop exporter round start (#1176)

34986db

* Use config to control noop exporter round start

Set pgsql exporter round from db on init (#1177)

959c576

Add waits/retries to algod importer (#1178)

a0b6ecc

* Add waits/retries to algod importer

conduit: pipeline run loop implementation (#1183)

9cc3d36

* conduit pipeline run loop implementation

Adds unit tests (#1202)

adbae7e

* Adds unit tests Resolves #1192 Adds unit tests to the config directory * Extra assert * PR comments * Wait for round increase * Added tests * context fix

succeed close calls when init fails (#1204)

bf8b441

silence usage data on error (#1205)

13aaba6

conduit: Conduit merge upstream (#1206)

d0046cb

Merge indexer/develop

conduit: Convert daemon to conduit pipeline (#1208)

88fca5c

* Convert indexer daemon to conduit

file exporter (#1213)

e8fa891

* adding file exporter

Conduit: e2e tests (#1221)

f93bd51

Add wait-for-block to block-gen (#1229)

756be9b

Conduit: prometheus metrics (#1230)

7cf43e8

* Add metrics reporting to conduit pipeline

Add filter_processor to all imports for processors (#1233)

a2e7bdc

File Exporter: save metadata more often (#1232)

ce54ff3

Conduit: init subcommand (#1235)

c809d35

Changed tag format (#1238)

3bc8f52

* Changed tag format * Updated generation code * Added filtering of numerical values * PR comments * imports

Add panic recovers (#1251)

a6b7473

* Add panic recovers Resolves #263 Adds recovery functions in main loop of conduit as well as in a go-routine in the start function of the pipeline * helper function

Condensed Duplicate logger output (#1256)

eaab3a1

Resolves #1245 Explicitly spell out empty data directory as well as condensed error messages when running conduit

AlgoStephenAkiki and others added 8 commits November 2, 2022 11:56

Added none operation (#1297)

5043497

Resolves #1283 Adds none operation to fields

Merge develop into conduit. (#1303)

964200e

Merge branch 'will/merge' into conduit

28ce7ca

Merge branch 'develop' into conduit

556e377

conduit: next-round command line flag (#1311)

9a2f836

List subcommand (#1296)

74a9954

Co-authored-by: Will Winder <[email protected]>

Minor pipeline test cleanup. (#1316)

9214b07

Reorganize plugin packages. (#1317)

c0e1146

winder requested review from mciccarello, Eric-Warehime, shiqizng and AlgoStephenAkiki November 10, 2022 23:09

winder added the Enhancement New feature or request label Nov 14, 2022

winder changed the title ~~Conduit~~ conduit: Initial conduit pipeline command. Nov 14, 2022

misc(release): add support for conduit (#1330)

b343cae

winder added the Team Lamprey label Nov 17, 2022

shiqizng reviewed Nov 17, 2022

View reviewed changes

conduit/plugins/exporters/noop/noop_exporter.go Outdated Show resolved Hide resolved

shiqizng reviewed Nov 17, 2022

View reviewed changes

conduit/plugins/exporters/example/example_exporter.go Outdated Show resolved Hide resolved

tzaffi and others added 4 commits November 25, 2022 08:17

New Feature: Application Boxes in Indexer (#1168)

75adc17

Enhancement: Update git submodule with go-algorand release commit `31…

837eadc

…a1099c` (#1304)

Adding a link to the data directory documentation (#1287)

ed7317e

Merge branch 'develop' into conduit

2cf6b2d

winder marked this pull request as ready for review November 29, 2022 18:46

Remove HandleGenesis references.

9f32987

Eric-Warehime approved these changes Nov 29, 2022

View reviewed changes

winder changed the title ~~conduit: Initial conduit pipeline command.~~ conduit: Initial conduit pipeline tool. Nov 29, 2022

winder merged commit d672efe into develop Nov 29, 2022

winder deleted the conduit branch November 29, 2022 19:35

AlgoStephenAkiki pushed a commit that referenced this pull request Jan 6, 2023

conduit: Initial conduit pipeline tool. (#1326)

5f6c66a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conduit: Initial conduit pipeline tool. #1326

conduit: Initial conduit pipeline tool. #1326

winder commented Nov 10, 2022 •

edited

Loading

codecov bot commented Nov 14, 2022 •

edited

Loading

conduit: Initial conduit pipeline tool. #1326

conduit: Initial conduit pipeline tool. #1326

Conversation

winder commented Nov 10, 2022 • edited Loading

Summary

Test Plan

codecov bot commented Nov 14, 2022 • edited Loading

Codecov Report

winder commented Nov 10, 2022 •

edited

Loading

codecov bot commented Nov 14, 2022 •

edited

Loading