Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conduit: filereader / filewriter plugins and compatibility changes. #1253

Merged
merged 39 commits into from
Oct 26, 2022

Conversation

winder
Copy link
Contributor

@winder winder commented Sep 30, 2022

Summary

  • New filereader plugin.
  • Small changes / improvements to the filewriter plugin:
    1. Add tags to the BlockData type so that the serialization is explicit.
    2. Change the block encoder function to use go-codec, the StateDelta object contains a map with non-string keys which caused a problem with the json encoder.
    3. Use t.TempDir() for tests so that the tests don't leave artifacts behind.
    4. Save a genesis.json file
    5. Some new configurations - Removing the vote certificate (should be in the filter plugin?), specifying the filename format.

Test Plan

New unit tests.

@codecov
Copy link

codecov bot commented Sep 30, 2022

Codecov Report

Merging #1253 (eb6631e) into conduit (99a848c) will increase coverage by 0.32%.
The diff coverage is 81.04%.

@@             Coverage Diff             @@
##           conduit    #1253      +/-   ##
===========================================
+ Coverage    58.39%   58.72%   +0.32%     
===========================================
  Files           77       78       +1     
  Lines        10332    10463     +131     
===========================================
+ Hits          6033     6144     +111     
- Misses        3772     3786      +14     
- Partials       527      533       +6     
Impacted Files Coverage Δ
data/block_export_data.go 82.69% <ø> (ø)
importer/helper.go 8.61% <0.00%> (-0.13%) ⬇️
util/ledger_util.go 62.50% <25.00%> (-3.41%) ⬇️
exporters/postgresql/postgresql_exporter.go 80.23% <40.00%> (-2.90%) ⬇️
exporters/filewriter/file_exporter.go 83.33% <82.60%> (+11.59%) ⬆️
util/util.go 76.92% <85.71%> (+10.25%) ⬆️
importers/filereader/filereader.go 85.93% <85.93%> (ø)
importers/algod/algod_importer.go 79.31% <100.00%> (+0.73%) ⬆️
processors/blockprocessor/block_processor.go 64.65% <100.00%> (+0.46%) ⬆️
processors/blockprocessor/initialize.go 63.88% <100.00%> (+0.33%) ⬆️
... and 2 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Comment on lines +68 to +71
err := os.MkdirAll(dataDir, 0755)
if err != nil {
return nil, fmt.Errorf("MakeProcessor() failed to create '%s': %w", dataDir, err)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatically create the local ledger directory if it's missing. While I was testing different configurations, this kept coming up.

@@ -185,8 +197,10 @@ func (exp *fileExporter) Round() uint64 {
return exp.round
}

func (exp *fileExporter) unmarhshalConfig(cfg string) error {
return yaml.Unmarshal([]byte(cfg), &exp.cfg)
func unmarshalConfig(cfg string) (Config, error) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contender for generics right here, we have the same function (with the same typo) in lots of places. I turned it into a pure function in a few places.

Comment on lines 184 to 187
genesisFilename := path.Join(exp.cfg.BlocksDir, "genesis.json")
if err := util.EncodeToFile(genesisFilename, genesis, true); err != nil {
return fmt.Errorf("HandleGenesis() failed to serialize genesis file: %w", err)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File reader needs the genesis file.

@winder winder marked this pull request as ready for review October 3, 2022 20:23
@winder winder changed the title Conduilt: filereader / filewriter plugins and compatibility changes. Conduit: filereader / filewriter plugins and compatibility changes. Oct 4, 2022
@shiqizng
Copy link
Contributor

LGTM

@winder winder merged commit 997b8c7 into algorand:conduit Oct 26, 2022
@winder winder deleted the will/filewriter-statedelta branch October 26, 2022 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants