add compressed mutable block type #1320
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a new block type that is mutable, allows random access, and tries to compress data as it is coming in. This should reduce the overhead for the data that is filling in and help avoid issue for out of order updates since the historical blocks will not be compressed to an immutable type.
brharrington
added a commit
to brharrington/atlas
that referenced
this pull request
Jun 9, 2021
In Netflix#1297, normalized values were buffered and would only get output at the end. This led to some user complaints that data was delayed. It was still within SLA, but it wouldn't show up until a bit later than before if the end time was set to now instead of offset by the SLA for data to be actionable. With the compressed mutable block type introduced in Netflix#1320, it is possible to update the older blocks as they are not immutable. In this change, the block store checks have been updated so the previous block can be updated to account for out of order values that may arrive. This should make the data visible earlier as it was before.
brharrington
added a commit
that referenced
this pull request
Jun 9, 2021
In #1297, normalized values were buffered and would only get output at the end. This led to some user complaints that data was delayed. It was still within SLA, but it wouldn't show up until a bit later than before if the end time was set to now instead of offset by the SLA for data to be actionable. With the compressed mutable block type introduced in #1320, it is possible to update the older blocks as they are not immutable. In this change, the block store checks have been updated so the previous block can be updated to account for out of order values that may arrive. This should make the data visible earlier as it was before.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new block type that is mutable, allows random access,
and tries to compress data as it is coming in. This should
reduce the overhead for the data that is filling in and help
avoid issue for out of order updates since the historical
blocks will not be compressed to an immutable type.