Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: twap key refactor #7472

Merged
merged 17 commits into from
Feb 17, 2024
Merged

feat: twap key refactor #7472

merged 17 commits into from
Feb 17, 2024

Conversation

czarcas7ic
Copy link
Member

@czarcas7ic czarcas7ic commented Feb 12, 2024

Closes: #7435

What is the purpose of the change

Previously, we required 2 different TWAP keys. One indexed by time, strictly used for pruning, and one indexed by the pool ID. We can refactor pruning logic to prune via the pool ID index. This allows us to reduce the 2x storage / deletion overhead.

Testing and Verifying

After implementing this change, the test that previously showed extra records being stored no longer shows these records being stored. All other tests show the same results as prior to this change.

Documentation and Release Note

  • Does this pull request introduce a new feature or user-facing behavior changes?
  • Changelog entry added to Unreleased section of CHANGELOG.md?

Where is the change documented?

  • Specification (x/{module}/README.md)
  • Osmosis documentation site
  • Code comments?
  • N/A

@github-actions github-actions bot added T:build C:app-wiring Changes to the app folder C:x/twap Changes to the twap module labels Feb 12, 2024
@czarcas7ic czarcas7ic added the V:state/breaking State machine breaking PR label Feb 12, 2024
@@ -168,7 +168,6 @@ go-mock-update:
mockgen -source=x/poolmanager/types/pool.go -destination=tests/mocks/pool.go -package=mocks
mockgen -source=x/gamm/types/pool.go -destination=tests/mocks/cfmm_pool.go -package=mocks
mockgen -source=x/concentrated-liquidity/types/cl_pool_extensionI.go -destination=tests/mocks/cl_pool.go -package=mocks
mockgen -source=ingest/sqs/domain/pools.go -destination=tests/mocks/sqs_pool.go -package=mocks -mock_names=PoolI=MockSQSPoolI
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drive by change, this folder no longer lives in Osmosis repo

x/twap/store.go Outdated Show resolved Hide resolved
@czarcas7ic czarcas7ic marked this pull request as ready for review February 13, 2024 21:42
Comment on lines +26 to +29
// Now that the TWAP keys are refactored, we can delete all time indexed TWAPs
// since we only need the pool indexed TWAPs.
keepers.TwapKeeper.DeleteAllHistoricalTimeIndexedTWAPs(ctx)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to selves, we may have to make these deletes smoothed over several blocks

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it depends on what else exists in the upgrade handler. If there are some other larger changes then yeah I think we should, otherwise itll be just like any other epoch, but less.

suite.Run(t, new(UpgradeTestSuite))
}

func (s *UpgradeTestSuite) TestUpgrade() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job on the test

var buffer bytes.Buffer
timeS := osmoutils.FormatTimeString(accumulatorWriteTime)
fmt.Fprintf(&buffer, "%s%s%s%d%s%s%s%s", HistoricalTWAPTimeIndexPrefix, timeS, KeySeparator, poolId, KeySeparator, denom1, KeySeparator, denom2)
fmt.Fprintf(&buffer, "%s%d%s%s%s%s", HistoricalTWAPPoolIndexPrefix, poolId, KeySeparator, denom1, KeySeparator, denom2)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to include the final KeySeparator!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why? The previous format did not have the final KeySeparator either

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prior one wasn't a prefix It was the full key

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, for anything that involves prefixes (not full key) it must include the final key separator, else iterations may include things that are just longer in the final key segment. (e.g. longer denom 1's) which could happen in pools with > 2 toknes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dang, we should have a CI for this or something

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added here e2fd17e

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we switch to collections / ORM, and quit constructing keys ourselves :)

Copy link
Member

@ValarDragon ValarDragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice job! I think it would be nice to separate out the pruning loop into something being in its own function, as its massive right now.

We have to do my key separator comment, otw we have a security bug edge case.

@ValarDragon
Copy link
Member

Can you update the README docs as well

CHANGELOG.md Outdated Show resolved Hide resolved
@github-actions github-actions bot added the C:docs Improvements or additions to documentation label Feb 16, 2024
Copy link
Member

@ValarDragon ValarDragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, this LGTM! Nice work.

Copy link
Member

@p0mvn p0mvn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

Reviewed core logic, and it LGTM! Did not review tests in-depth.

@ValarDragon
Copy link
Member

I think we can merge once tests pass

@ValarDragon ValarDragon merged commit 49cd790 into main Feb 17, 2024
1 check passed
@ValarDragon ValarDragon deleted the adam/twap-key-refactor branch February 17, 2024 04:01
@github-actions github-actions bot mentioned this pull request Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C:app-wiring Changes to the app folder C:docs Improvements or additions to documentation C:x/twap Changes to the twap module T:build V:state/breaking State machine breaking PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Restructure TWAP pruning keys
3 participants