Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deps: add simdutf dependency #45803

Closed
wants to merge 3 commits into from
Closed

deps: add simdutf dependency #45803

wants to merge 3 commits into from

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Dec 9, 2022 •
edited
Loading

simdutf provides a faster way of providing utf8 operations with SIMD instructions. @nodejs/undici team was looking for a way to validate utf8 input, and this dependency can make it happen.

Edit: I'm proposing either exposing the following functionality through a new module (like node:encoding) or through util.types or buffer

  • validate_ascii(string)
  • validate_utf8(string)
  • count_utf8(string)

PS: simdutf supports more features, and depending on the need, it makes more sense to expose them through a new module, instead of util.types or buffer.

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/gyp

@nodejs-github-bot nodejs-github-bot added build Issues and PRs related to build files or the CI. dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. tools Issues and PRs related to the tools directory. labels Dec 9, 2022
@anonrig anonrig mentioned this pull request Dec 9, 2022
Merged
12 tasks
@anonrig anonrig force-pushed the deps/simdutf branch 2 times, most recently from 2c20c9a to 6daa546 Compare December 9, 2022 21:09
@anonrig anonrig changed the title dep: add simdutf dependency deps: add simdutf dependency Dec 9, 2022
@KhafraDev
Copy link
Member

This would help speedup both ws and undici's WebSocket implementation (which is still WIP). When we receive a text frame or receive a close frame with a reason, we need to validate that the buffer contains valid utf-8.

There are a few ways of doing so currently: a js implementation by default in both undici and ws, and optionally a package such as utf-8-validate. Note that simdutf is many times faster than the c++ version of utf-8-validate in the benchmark above, and the js fallback version is the slowest.

Here is a PR from @lpinca that shows massive speedups when using simdutf: websockets/utf-8-validate#101. Considering how widespread usage of ws is, exposing a very fast ability to validate utf-8 would improve a ton of the ecosystem.

@anonrig anonrig force-pushed the deps/simdutf branch 2 times, most recently from 5027cae to e94ba5f Compare December 9, 2022 21:29
@richardlau richardlau added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 9, 2022
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 9, 2022
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48372/

@anonrig anonrig force-pushed the deps/simdutf branch 2 times, most recently from bed88cc to 4269faf Compare December 9, 2022 23:26
mscdex
mscdex reviewed Dec 9, 2022
View reviewed changes
deps/simdutf/simdutf.gyp Outdated Show resolved Hide resolved
@github-actions github-actions bot mentioned this pull request Dec 10, 2022
Open
9 tasks
@anonrig anonrig force-pushed the deps/simdutf branch 3 times, most recently from ced7ef2 to 5566c99 Compare December 10, 2022 02:53
@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48377/

@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
fd77454 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
79cf6bc 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
85a26c6 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
3d55e4b 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 3, 2023
@RafaelGSS
2023-01-04, Version 19.4.0 (Current)
ef55c9c 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @RafaelGSS
deps: add simdutf dependency
f537eac 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @RafaelGSS
util: add fast path for text-decoder fatal flag
73209f0 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @lemire @RafaelGSS
deps: disable avx512 for simutf on benchmark ci
1f889ae 
Co-authored-by: Daniel Lemire <[email protected]>
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
980a9ca 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
6e0a7be 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
1ed0ef2 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
790cc5b 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @RafaelGSS
deps: add simdutf dependency
344c5ec 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @RafaelGSS
util: add fast path for text-decoder fatal flag
ae842a4 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @lemire @RafaelGSS
deps: disable avx512 for simutf on benchmark ci
98fc94a 
Co-authored-by: Daniel Lemire <[email protected]>
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
b181545 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
e58ed6d 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 6, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
22a2ec6 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
juanarbol pushed a commit that referenced this pull request Jan 26, 2023
@anonrig @juanarbol
deps: add simdutf dependency
e23ab5d 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
juanarbol pushed a commit that referenced this pull request Jan 26, 2023
@anonrig @juanarbol
util: add fast path for text-decoder fatal flag
96fc1ca 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
juanarbol pushed a commit that referenced this pull request Jan 26, 2023
@anonrig @lemire @juanarbol
deps: disable avx512 for simutf on benchmark ci
dac0f2e 
Co-authored-by: Daniel Lemire <[email protected]>
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
de266a4 
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: TBD
@juanarbol juanarbol mentioned this pull request Jan 28, 2023
Merged
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
85ed13d 
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 28, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
614a60d 
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
juanarbol added a commit that referenced this pull request Jan 30, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
53ddd3a 
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
@anonrig @juanarbol
deps: add simdutf dependency
c9845fc 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
@anonrig @juanarbol
util: add fast path for text-decoder fatal flag
f1dcbe7 
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
@anonrig @lemire @juanarbol
deps: disable avx512 for simutf on benchmark ci
64d3e3f 
Co-authored-by: Daniel Lemire <[email protected]>
PR-URL: #45803
Reviewed-By: Robert Nagy <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Michael Dawson <[email protected]>
juanarbol added a commit that referenced this pull request Jan 31, 2023
@juanarbol
2023-01-31, Version 18.14.0 'Hydrogen' (LTS)
a413280 
Notable changes:

* buffer
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
* deps:
  * add simdutf dependency (Yagiz Nizipli) #45803
  * upgrade npm to 9.1.3 (npm team) #45693
* util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46396
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Uzlopak Uzlopak Uzlopak left review comments

@targos targos targos left review comments

@lpinca lpinca lpinca left review comments

@lemire lemire lemire left review comments

@mhdawson mhdawson mhdawson approved these changes

@addaleax addaleax addaleax approved these changes

@mcollina mcollina mcollina approved these changes

@mscdex mscdex mscdex left review comments

@ronag ronag Awaiting requested review from ronag

@daeyeon daeyeon Awaiting requested review from daeyeon

@bnoordhuis bnoordhuis Awaiting requested review from bnoordhuis

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. build Issues and PRs related to build files or the CI. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. performance Issues and PRs related to the performance of Node.js. review wanted PRs that need reviews. tools Issues and PRs related to the tools directory.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.