Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use array instead of int map for itemTags #1357

Merged
merged 1 commit into from
Oct 7, 2021

Conversation

brharrington
Copy link
Contributor

The itemTags field of the index is only used for findKeys
and findValues. In the findKeys case it loops over all keys
and the simple array is faster. For the findValues case, it
gets the value for a key which is now linear for the number
of tags so will be a bit slower.

The main benefit is it can reduce the memory use considerably
for large indexes. The IntIntHashMap has an overhead of 40
bytes and also allocates an array of 2x the size to avoid
collisions in the map. With a sample index of 10M items this
change reduced the memory size by several GB.

The itemTags field of the index is only used for findKeys
and findValues. In the findKeys case it loops over all keys
and the simple array is faster. For the findValues case, it
gets the value for a key which is now linear for the number
of tags so will be a bit slower.

The main benefit is it can reduce the memory use considerably
for large indexes. The IntIntHashMap has an overhead of 40
bytes and also allocates an array of 2x the size to avoid
collisions in the map. With a sample index of 10M items this
change reduced the memory size by several GB.
@brharrington brharrington added this to the 1.7.0 milestone Oct 7, 2021
@brharrington brharrington merged commit 2309398 into Netflix:master Oct 7, 2021
@brharrington brharrington deleted the idx-itemtags branch October 7, 2021 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant