Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation for data directory. #1125

Merged
merged 7 commits into from
Jul 22, 2022

Conversation

winder
Copy link
Contributor

@winder winder commented Jul 18, 2022

Summary

Some documentation about the new data directory.

@codecov
Copy link

codecov bot commented Jul 18, 2022

Codecov Report

Merging #1125 (2fc0149) into release/2.13.0 (30b60c2) will increase coverage by 0.03%.
The diff coverage is n/a.

@@                Coverage Diff                 @@
##           release/2.13.0    #1125      +/-   ##
==================================================
+ Coverage           60.05%   60.09%   +0.03%     
==================================================
  Files                  52       52              
  Lines                8750     8750              
==================================================
+ Hits                 5255     5258       +3     
+ Misses               3012     3010       -2     
+ Partials              483      482       -1     
Impacted Files Coverage Δ
fetcher/fetcher.go 52.19% <0.00%> (+1.31%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us.

@winder winder self-assigned this Jul 19, 2022

Indexer writers keep an account cache in the data directory. This cache is used during block processing to compute things like the new account balances after processing transactions. Prior to this local cache, the database was queried on each round to fetch the initial account states.

## Read-Only Mode
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If an indexer is started in writer mode and then is re-started in reader mode, will these files be there or will they be deleted by the indexer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They wouldn't be deleted, but wouldn't be updated either.

docs/DataDirectory.md Outdated Show resolved Hide resolved
@winder winder marked this pull request as ready for review July 19, 2022 21:16
docs/DataDirectory.md Outdated Show resolved Hide resolved
Copy link
Contributor

@algobarb algobarb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have someone from Product or Dev/Rel review these docs too to make sure they understand the new changes?

docs/DataDirectory.md Outdated Show resolved Hide resolved
docs/DataDirectory.md Outdated Show resolved Hide resolved
@winder winder requested a review from chaihoang July 21, 2022 18:20
Copy link
Contributor

@algobarb algobarb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

docs/DataDirectory.md Outdated Show resolved Hide resolved
docs/DataDirectory.md Outdated Show resolved Hide resolved
@winder winder merged commit 996bf10 into algorand:release/2.13.0 Jul 22, 2022
@winder winder deleted the will/data-dir-doc branch July 22, 2022 15:01
Eric-Warehime added a commit that referenced this pull request Aug 1, 2022
* Bump version to 2.13.0-rc1

* Bump version to 2.13.0

* Documentation for data directory. (#1125)

Co-authored-by: algobarb <[email protected]>

* Don't lookup big foreign assets. (#1141)

* Revert "Bump version to 2.13.0"

This reverts commit 0a8af61.

* Bump version to 2.13.0

* Fix import performance test runner. (#1133)

* Start on round 1 since round 0 is now computed from the genesis file.
* Wait for indexer processor to exit.
* Better logging for metric collection errors.
* Proper support for data directory.
* New test script for future release automation.

* Revert "Bump version to 2.13.0"

This reverts commit 7915890.

* Bump version to 2.13.0

* test fixes: Submodule updates (#1144)

* Update go-algorand submodule
* Fix test failure due to duplicate txns
* Add new ledger interface method

* Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146)

removing a bunch of code and make the random test pass with the new ledger for evaluator

* Docs: Readme update (#1149)

* Update README header

* Testing: Use tempdir instead of /tmp for e2elive test (#1152)

* Format misc/*.py with `black` (#1153)

* apply black to e2elive.py as well (#1154)

* Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151)

* Eric's Dockerfile improvements
* Update misc/e2elive.py

Co-authored-by: DevOps Service <[email protected]>
Co-authored-by: Will Winder <[email protected]>
Co-authored-by: algobarb <[email protected]>
Co-authored-by: Barbara Poon <[email protected]>
Co-authored-by: Zeph Grunschlag <[email protected]>
Eric-Warehime added a commit that referenced this pull request Sep 14, 2022
* Bump version to 2.13.0-rc1

* Bump version to 2.13.0

* Documentation for data directory. (#1125)

Co-authored-by: algobarb <[email protected]>

* Don't lookup big foreign assets. (#1141)

* Revert "Bump version to 2.13.0"

This reverts commit 0a8af61.

* Bump version to 2.13.0

* Fix import performance test runner. (#1133)

* Start on round 1 since round 0 is now computed from the genesis file.
* Wait for indexer processor to exit.
* Better logging for metric collection errors.
* Proper support for data directory.
* New test script for future release automation.

* Revert "Bump version to 2.13.0"

This reverts commit 7915890.

* Bump version to 2.13.0

* test fixes: Submodule updates (#1144)

* Update go-algorand submodule
* Fix test failure due to duplicate txns
* Add new ledger interface method

* Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146)

removing a bunch of code and make the random test pass with the new ledger for evaluator

* Docs: Readme update (#1149)

* Update README header

* Testing: Use tempdir instead of /tmp for e2elive test (#1152)

* Format misc/*.py with `black` (#1153)

* apply black to e2elive.py as well (#1154)

* Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151)

* Eric's Dockerfile improvements
* Update misc/e2elive.py

* Bug-Fix: Implement BlockHdrCached + miscellany (#1162)

* Enhancement: add max int64 checks (#1166)

* state proofs: Indexer Support for State Proofs (#1002)

Adds API support to the Indexer for State Proof Transactions and header fields.

Co-authored-by: Will Winder <[email protected]>

Co-authored-by: Will Winder <[email protected]>

* Bump version to 2.14.0-rc1

* Stop Panics if no config is supplied (#1180)

Give a default config if not supplied to stop panics.

* Fix spec name collisions. (#1182)

* Update go-algorand submodule to v3.9.1-beta (#1185)

* Bump version to 2.14.0-rc2

* Disable deadlock detection (#1186)

* Add support for new block header: TxnRoot SHA256 (#989)

* Accept yaml and yml configuration files. (#1181)

* Fix bug in reveals lookup (#1198)

* Fix bug in reveals lookup (#1198)

* Bump version to 2.14.0-rc3

* add state proof example with high reveal index - from betanet (#1199)

* Devops: Bump go-algorand submodule to v3.9.2-beta (#1203)

* Bump version to 2.14.0-rc4

* enhancement: Clarify REST query parameters for accounts search (#1201)

* update description for /v2/accounts

* cicd: add darwin arm64 support to release script (#1169)

* Bump version to 2.14.0

* Downgrade mockery to prevent incorrect deprecation warning. (#1211)

* Enhancement: update e2e test policy (#1197)

*update e2e test policy

* Fix release 2.14.0 (#1214)

* Accept yaml and yml configuration files. (#1181)

* Fix bug in reveals lookup (#1198)

* add state proof example with high reveal index - from betanet (#1199)

* enhancement: Clarify REST query parameters for accounts search (#1201)

* update description for /v2/accounts

* cicd: add darwin arm64 support to release script (#1169)

* Downgrade mockery to prevent incorrect deprecation warning. (#1211)

* Enhancement: update e2e test policy (#1197)

*update e2e test policy

* Update test expected value: transaction root sha256

Co-authored-by: AlgoStephenAkiki <[email protected]>
Co-authored-by: Michael Diamant <[email protected]>
Co-authored-by: algoidan <[email protected]>
Co-authored-by: shiqizng <[email protected]>
Co-authored-by: algolucky <[email protected]>
Co-authored-by: Will Winder <[email protected]>

Co-authored-by: DevOps Service <[email protected]>
Co-authored-by: Will Winder <[email protected]>
Co-authored-by: algobarb <[email protected]>
Co-authored-by: Barbara Poon <[email protected]>
Co-authored-by: Zeph Grunschlag <[email protected]>
Co-authored-by: shiqizng <[email protected]>
Co-authored-by: AlgoStephenAkiki <[email protected]>
Co-authored-by: John Lee <[email protected]>
Co-authored-by: Or Aharonee <[email protected]>
Co-authored-by: Michael Diamant <[email protected]>
Co-authored-by: algoidan <[email protected]>
Co-authored-by: algolucky <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants