test fixes: Submodule updates #1144

Eric-Warehime · 2022-07-25T20:27:03Z

Summary

A new field in Teal has shown up in the e2e test files which means that we need to update the submodule. However, because the submodule update also pulls in the 320 round changes, there are some test failures showing up.

The root cause/fix for these is that each transaction submitted to a ledger needs to have a different checksum. In order to accomplish this I've put a "random" byte array into the notes field of each transaction created, and have re-created transactions instead of reusing them in our tests.

I've also added a new LedgerForEvaluator method since that was added the upstream interface as well. It is a very simple wrapper method.

Test Plan

CI

codecov · 2022-07-25T20:37:38Z

Codecov Report

Merging #1144 (5307cbf) into develop (54d39d7) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop    #1144      +/-   ##
===========================================
- Coverage    60.09%   60.04%   -0.05%     
===========================================
  Files           52       52              
  Lines         8750     8752       +2     
===========================================
- Hits          5258     5255       -3     
- Misses        3010     3014       +4     
- Partials       482      483       +1

Impacted Files	Coverage Δ
processor/eval/ledger_for_evaluator.go	`75.55% <0.00%> (-1.72%)`	⬇️
fetcher/fetcher.go	`50.87% <0.00%> (-1.32%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us.

tzaffi · 2022-07-25T20:41:22Z

util/test/account_testutil.go

+func ArbitraryString() []byte {
+	arb := make([]byte, config.MaxTxnNoteBytes)
+	rand.Read(arb)
+	return arb


tzaffi · 2022-07-25T20:51:28Z

processor/eval/ledger_for_evaluator.go

@@ -148,3 +148,8 @@ func (l LedgerForEvaluator) LatestTotals() (ledgercore.AccountTotals, error) {
 	_, totals, err := l.Ledger.LatestTotals()
 	return totals, err
 }
+
+// BlockHdrCached is part of go-algorand's indexerLedgerForEval interface.


I don't see where else in the PR this function is used. Was this brought in by the go-algorand update? Is there a way to know that this is implemented correctly (seems like it should since this is a pass-through) ?

I should also mention that if indexer is not actually making use of this method, I would argue for implementing via panic("not implemented") instead.

Yeah it was added in algorand/go-algorand@b8007e4#diff-69a9375fb82e9e6878855f6e962f7b3eeba8bf108dc6c2c07ab21e35cfb8d267R47

And no we don't call it in our code, but it has to be defined as part of the interface, so I just copied what I saw in the above commit.

I'm actually not confident that it's not used. It seems to be called inside eval in b8007e4ade943d4bc181a301d1291685eaf60fc4 and as you've pointed out, there wouldn't seem to be a reason to add it to the interface if it wasn't being used.

tzaffi

The changes all make sense, but I had a question regarding the BlockHdrCached method in processor/eval/ledger_for_evaluator.go.

tzaffi

LGTM and 🤞 that CI will agree

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <[email protected]> Co-authored-by: Will Winder <[email protected]> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: algolucky <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: John Lee <[email protected]> Co-authored-by: Or Aharonee <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: algolucky <[email protected]>

Eric-Warehime added 4 commits July 25, 2022 13:22

Update submodule

54542f2

Update ledger for eval definition

b832cb6

Add comment

204c474

Add random Note to Txns in tests

25010ba

Eric-Warehime added the Bug-Fix label Jul 25, 2022

Eric-Warehime requested review from winder, tzaffi, shiqizng and AlgoStephenAkiki July 25, 2022 20:27

shiqizng approved these changes Jul 25, 2022

View reviewed changes

tzaffi reviewed Jul 25, 2022

View reviewed changes

Remove implementation of unused method

5307cbf

tzaffi approved these changes Jul 25, 2022

View reviewed changes

Eric-Warehime merged commit ed2a991 into algorand:develop Jul 25, 2022

Eric-Warehime deleted the submodule-updates branch July 25, 2022 21:54

This was referenced Aug 3, 2022

New Feature: Indexer Application Box Ingestion #1067

Merged

Bug-Fix: Implement BlockHdrCached + miscellany #1162

Merged

This was referenced Aug 10, 2022

FOR REVIEW ONLY: indexer 2.14.0 into master #1170

Merged

FOR REVIEW ONLY: indexer 2.14.0 into develop #1171

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test fixes: Submodule updates #1144

test fixes: Submodule updates #1144

Eric-Warehime commented Jul 25, 2022

codecov bot commented Jul 25, 2022 •

edited

Loading

tzaffi Jul 25, 2022

tzaffi Jul 25, 2022

tzaffi Jul 25, 2022

Eric-Warehime Jul 25, 2022

Eric-Warehime Jul 25, 2022

tzaffi left a comment

tzaffi left a comment

test fixes: Submodule updates #1144

test fixes: Submodule updates #1144

Conversation

Eric-Warehime commented Jul 25, 2022

Summary

Test Plan

codecov bot commented Jul 25, 2022 • edited Loading

Codecov Report

tzaffi Jul 25, 2022

Choose a reason for hiding this comment

tzaffi Jul 25, 2022

Choose a reason for hiding this comment

tzaffi Jul 25, 2022

Choose a reason for hiding this comment

Eric-Warehime Jul 25, 2022

Choose a reason for hiding this comment

Eric-Warehime Jul 25, 2022

Choose a reason for hiding this comment

tzaffi left a comment

Choose a reason for hiding this comment

tzaffi left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 25, 2022 •

edited

Loading