feat: use TextEncoder and TextDecoder for utf8 strings
#4513
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
master, af1a9c9 seia-soto:textencoder, 1de005e |
chrmod
reviewed
Dec 11, 2024
chrmod
approved these changes
Dec 11, 2024
philipp-classen
approved these changes
Dec 11, 2024
seia-soto
commented
Dec 11, 2024
remusao
reviewed
Dec 11, 2024
- ~65535 ASCII only characters
> 147.50420889870574 / 156.1296767089117 // benchEngineDeserialization
0.944754463135876
> 147.50420889870574 / 148.7394726802865 // benchEngineSerialization
0.9916951179177841seia-soto:textencoder, 65764e9
master, de7bfb5
|
remusao
reviewed
Dec 12, 2024
packages/adblocker/src/data-view.ts
Outdated
Comment on lines
394
to
395
| const { written } = TEXT_ENCODER.encodeInto(raw, this.buffer.subarray(this.pos + 4)); | ||
| this.pushLength(written); |
There was a problem hiding this comment.
Since you do +4 here (32 bits), it's not useful to use pushLength since you might as well push the length always as uint32. The only reason to use pushLength is to use less bits to encode the length of small strings.
There was a problem hiding this comment.
Yup, I'm on the fix to support dynamic positioning to utilize pushLength.
There was a problem hiding this comment.
@remusao Please, review 0339bdc (#4513)
There was a problem hiding this comment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #4424
This PR replaces punycode encoder and decoder with
TextEncoderandTextDecoderfor utf8 strings.\ufeffshould be skipped when decoding to ensure the original formUint8Array.subarraydoesn't copy the array but provides a direct interface to subarrayTextEncoder.encodeIntodoesn't produce EOL character ( NULL, U+0000 ) but we don't care becauseTextDecodercan stop nicely when provided buffer endsTo be safe: