Add support for new block header: TxnRoot SHA256 #989

Aharonee · 2022-05-03T16:30:43Z

No description provided.

codecov-commenter · 2022-05-03T16:44:22Z

Codecov Report

Merging #989 (a84d77e) into release/2.14.0 (b129605) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           release/2.14.0     #989   +/-   ##
===============================================
  Coverage           60.81%   60.81%           
===============================================
  Files                  52       52           
  Lines                8475     8476    +1     
===============================================
+ Hits                 5154     5155    +1     
  Misses               2862     2862           
  Partials              459      459

Impacted Files	Coverage Δ
api/generated/common/routes.go	`46.51% <ø> (ø)`
api/generated/v2/routes.go	`17.76% <ø> (ø)`
api/handlers.go	`75.18% <100.00%> (+0.02%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

winder

This looks fine, but lets keep the PR open until we can update the submodule and handle the TODO (I see now that this PR is a draft, so that was probably the plan already).

Eric-Warehime · 2022-05-13T16:20:07Z

Picking up this PR since Or already did some of the work and we're ready to get this stuff committed to indexer because it's already in go-algorand master.

Eric-Warehime · 2022-05-16T15:35:06Z

@Aharonee @algoidan Would be great to get your feedback on these changes.

Aharonee · 2022-05-16T15:51:47Z

@Aharonee @algoidan Would be great to get your feedback on these changes.

Looks good to me

bertani · 2022-05-17T12:16:42Z

Just a note about the fact that while the mainnet block production has already switched to block headers containing the new sha256 txnroot field few days ago, the indexer still not having it can cause serious difficulties to external systems who need to independently verify block validation. I hope this gets merged and released soon.

Aharonee · 2022-05-17T12:59:44Z

@bertani This change to the BlockHeader is still under ConsensusFuture so it hasn't taken affect on the mainnet as of yet.
https://github.com/algorand/go-algorand/blob/91095fa6bdccb11ec133c38da68fdd92fa611b6c/config/consensus.go#L1152

bertani · 2022-05-17T14:35:24Z

@Aharonee then my bad, I had assumed that single the block header calculation started failing over the last week, it was due to this change, it must be for a different reason then

id-ms

LGTM

have some minor comments

api/indexer.oas3.yml

id-ms · 2022-05-19T14:09:41Z

api/generated/common/types.go

@@ -339,6 +339,9 @@ type Block struct {
 	// \[txn\] TransactionsRoot authenticates the set of transactions appearing in the block. More specifically, it's the root of a merkle tree whose leaves are the block's Txids, in lexicographic order. For the empty block, it's 0. Note that the TxnRoot does not authenticate the signatures on the transactions, only the transactions themselves. Two blocks with the same transactions but in a different order and with different signatures will have the same TxnRoot.
 	TransactionsRoot []byte `json:"transactions-root"`

+	// \[txn256\] TransactionsRootSHA256 is an auxiliary TransactionRoot, built using a vector commitment instead of a merkle tree, and SHA256 hash function instead of the default SHA512_256. It is used mostly for StateProofs.
+	TransactionsRootSha256 []byte `json:"transactions-root-sha256"`


should we give TransactionsRootSha256 and TransactionsRoot the same names as in algod?

I don't think algod defines the fields for a block--instead preferring to leave them as simply an opaque object which is returned. So in that sense these fields would not exist for algod to my knowledge.

That field is not defined in algod as is, but it directly corresponds to the new fields in the BlockHeader.
I chose not to rename these fields (as I've renamed TxnRoot -> TxnCommitments struct) to stay backwards compatible to the previously defined type.

We could rename them and keep the json keys as they are though;

TransactionsRoot -> TransactionCommitmentsNativeSha512_256 TransactionsRootSha256 -> TransactionCommitmentsSha256

There is a field in the code generator which is supposed to allow us to diverge the json from the generated class name. However, it doesn't seem to be respected by the code generator when I change/run it, and also having the two diverge in that way seems confusing from a user perspective.

I think the path forward I'd like to take here is to just maintain the existing transactions-root and add in the transactions-root-shaa256 as we are currently doing here.

winder · 2022-08-02T17:51:05Z

Closing in favor of #1002

# Conflicts: # api/generated/common/routes.go # api/generated/v2/routes.go # api/handlers.go

Eric-Warehime · 2022-08-18T17:27:17Z

Need to update https://github.com/algorand/indexer/blob/develop/api/test_resources/state_proof.response to include the expected changes.

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <[email protected]> Co-authored-by: Will Winder <[email protected]> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: algolucky <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: John Lee <[email protected]> Co-authored-by: Or Aharonee <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: algolucky <[email protected]>

Add support for new block header: TxnRoot SHA256

5b427de

Aharonee added the Enhancement New feature or request label May 3, 2022

Aharonee requested review from winder and id-ms May 3, 2022 16:30

winder reviewed May 9, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/develop' into txnroot-sha256

57551b3

Eric-Warehime marked this pull request as ready for review May 13, 2022 16:55

Eric-Warehime requested a review from winder May 13, 2022 16:55

chaihoang added the Team Lamprey label May 13, 2022

Eric-Warehime self-assigned this May 17, 2022

id-ms approved these changes May 19, 2022

View reviewed changes

Update description for txnrootsha256

40d8d57

chaihoang removed the Team Lamprey label May 31, 2022

winder closed this Aug 2, 2022

Aharonee reopened this Aug 18, 2022

Merge branch 'develop' into txnroot-sha256

19d625f

# Conflicts: # api/generated/common/routes.go # api/generated/v2/routes.go # api/handlers.go

winder changed the base branch from develop to release/2.14.0 August 18, 2022 16:33

winder added 2 commits August 18, 2022 12:34

Merge branch 'release/2.14.0' into txnroot-sha256

b8c0b97

Fix fmt

10387ba

winder approved these changes Aug 18, 2022

View reviewed changes

Eric-Warehime approved these changes Aug 18, 2022

View reviewed changes

Fix test.

a84d77e

winder merged commit 055a502 into release/2.14.0 Aug 18, 2022

winder deleted the txnroot-sha256 branch August 18, 2022 20:29

This was referenced Aug 19, 2022

FOR REVIEW ONLY: indexer 2.14.0 into master #1170

Merged

FOR REVIEW ONLY: indexer 2.14.0 into develop #1171

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for new block header: TxnRoot SHA256 #989

Add support for new block header: TxnRoot SHA256 #989

Aharonee commented May 3, 2022 •

edited

Loading

codecov-commenter commented May 3, 2022 •

edited by codecov bot

Loading

winder left a comment •

edited

Loading

Eric-Warehime commented May 13, 2022

Eric-Warehime commented May 16, 2022

Aharonee commented May 16, 2022

bertani commented May 17, 2022 •

edited

Loading

Aharonee commented May 17, 2022

bertani commented May 17, 2022

id-ms left a comment

id-ms May 19, 2022

Eric-Warehime May 23, 2022

Aharonee May 24, 2022

Aharonee May 24, 2022

Eric-Warehime May 27, 2022

winder commented Aug 2, 2022

Eric-Warehime commented Aug 18, 2022

Add support for new block header: TxnRoot SHA256 #989

Add support for new block header: TxnRoot SHA256 #989

Conversation

Aharonee commented May 3, 2022 • edited Loading

codecov-commenter commented May 3, 2022 • edited by codecov bot Loading

Codecov Report

winder left a comment • edited Loading

Choose a reason for hiding this comment

Eric-Warehime commented May 13, 2022

Eric-Warehime commented May 16, 2022

Aharonee commented May 16, 2022

bertani commented May 17, 2022 • edited Loading

Aharonee commented May 17, 2022

bertani commented May 17, 2022

id-ms left a comment

Choose a reason for hiding this comment

id-ms May 19, 2022

Choose a reason for hiding this comment

Eric-Warehime May 23, 2022

Choose a reason for hiding this comment

Aharonee May 24, 2022

Choose a reason for hiding this comment

Aharonee May 24, 2022

Choose a reason for hiding this comment

Eric-Warehime May 27, 2022

Choose a reason for hiding this comment

winder commented Aug 2, 2022

Eric-Warehime commented Aug 18, 2022

Aharonee commented May 3, 2022 •

edited

Loading

codecov-commenter commented May 3, 2022 •

edited by codecov bot

Loading

winder left a comment •

edited

Loading

bertani commented May 17, 2022 •

edited

Loading