Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try out caching and ASCII special casing for other segmenters #80

Open
Manishearth opened this issue Feb 14, 2020 · 0 comments
Open

Try out caching and ASCII special casing for other segmenters #80

Manishearth opened this issue Feb 14, 2020 · 0 comments

Comments

@Manishearth
Copy link
Member

Manishearth commented Feb 14, 2020

@cessen in #77 and #79 implemented a complementary pair of optimizations for grapheme segmentation: one which optimizes the binary search to stay in the found region, and one which handles ASCII cases. They work well together, even for non-Latin text, because the caching is much more efficient when common punctuation/spaces are handled directly and do not invalidate the cache.

It might be worth doing the same thing for word, line, and sentence breaking. In some of these cases a table lookup instead of if branches may work out better, though I think if branches will typically be faster.

It may also be worth tweaking the binary search so that even if the character is not in the cached range, it can search nearby first. This is useful for e.g. Indic scripts, where you're going to have a rapid mix of categories, but all the code points are nearby.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant