Jest 24 is 67% slower than Jest 23.6.0 on the same test suite #7732

joelgetaction · 2019-01-28T04:19:18Z

🐛 Bug Report

Thanks for your hard work releasing jest 24!

A clear and concise description of what the bug is.

On jest 23.6.0 my test collection that totals 49 test suites and 700 tests took on average 11.9 seconds to run. Now in jest 24.0.0 it takes 19.9 seconds to run, with no changes of any kind to my code, tests or environment other than upgrading to jest 24. I'm not doing anything fancy.

This is a serious performance regression. I want to use jest 24 to take advantage of test.todo but I can't afford to have my tests take 67% longer to run - that's a real drop in my productivity.

Any suggestions how to diagnose this further or fix this issue? I can't post source code because this is proprietary code I'm working on. How can we fix this performance regression?

To Reproduce

Steps to reproduce the behavior:

Run a set of tests in jest 23.6.0.
Run that same set of tests in jest 24.0.0.

Expected behavior

A clear and concise description of what you expected to happen.

Jest 24.0.0 should NOT be 67% slower than jest 23.6.0 for the same set of tests.

Link to repl or repo (highly encouraged)

Please provide either a repl.it demo or a minimal repository on GitHub.

Issues without a reproduction link are likely to stall.

Run `npx envinfo --preset jest`

Paste the results here:

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (4) x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  Binaries:
    Node: 10.15.0 - ~/.nvm/versions/node/v10.15.0/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v10.15.0/bin/npm
  npmPackages:
    jest: ^24.0.0 => 24.0.0

The text was updated successfully, but these errors were encountered:

SimenB · 2019-01-28T06:31:24Z

We need some sort of reproduction here. I haven't seen any difference between 23 and 24 when upgrading, so it might be certain patterns that hsve regressed

CrOrc · 2019-01-28T07:00:05Z

For me the Jest keeps writing new jest-transform-cache folder on every jest run.

System:
OS: Windows 7
CPU: (8) x64 Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
Binaries:
Node: 11.7.0 - C:\Program Files\nodejs\node.EXE
Yarn: 1.13.0 - ~\scoop\apps\yarn\current\Yarn\bin\yarn.CMD
npm: 6.5.0 - C:\Program Files\nodejs\npm.CMD

SimenB · 2019-01-28T07:07:30Z

Interesting! That would definitely slow things down. Does that happen in many projects? Could you post your config?

CrOrc · 2019-01-28T07:15:47Z

Here are babel.config.js, jest.config.js, package.json, yarn.lock

jest.config.zip

lirbank · 2019-01-28T17:41:18Z

To me 24 seems slower when there is a cache available, but about the same as 23 when there is no cache, so it seems like @CrOrc is on to something. Here are my results (ran the tests three times with each config):

23.6.0 - Runs after clearing the cache with --clearCache:

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 64.927s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 54.173s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 59.353s

23.6.0 - Runs without clearing the cache:

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 36.196s, estimated 60s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 32.83s, estimated 33s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 32.278s

24.0.0 - Runs after clearing the cache with --clearCache:

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 56.718s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 64.682s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 61.646s

24.0.0 - Runs without clearing the cache:

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 42.998s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 44.611s

Test Suites: 36 passed, 36 total
Tests: 553 passed, 553 total
Snapshots: 48 passed, 48 total
Time: 48.576s

overlookmotel · 2019-01-29T15:03:51Z

Just to say that I'm seeing the same as @CrOrc. Every run of jest with 24.0.0 produces a new jest-transform-cache folder plus new perf-cache and haste-map. Hence every test run takes as long as the first - the cache seems to be ignored. This doesn't happen with jest 23.6.0.

csvan · 2019-01-29T15:11:14Z

Cannot provide repro due to working on proprietary code, but I can confirm this from our CI logs. Tests phase takes up to 5 minutes longer after moving from Jest 23.6 -> 24.0, with no other code changes being made.

I tried adding the --no-cache flag to see if it could be cache related as per @CrOrc suggestion above. I saw no difference in runtime. I then investigated I/O activity differences between 23.6 and 24.0 and found quite a significant difference:

Jest 24.0 : 10-15 MB/s total for 4 Jest workers
Jest 23.6: 1-3 MB/s total for 4 Jest workers

I know the above is anecdotal for our machines, but as suggested by @CrOrc it seems that Jest 24.0 makes significantly more cache writes than 23.6.

SimenB · 2019-01-29T15:43:15Z

I can confirm caching of transformed files are broken.

It's due to this line (from #5862): https://github.com/facebook/jest/blob/31b81ba37eb779161b6db78aa862c0bcd6465bd5/packages/jest-config/src/normalize.js#L272

Removing it speeds up rerunning the same tests by 300% for me in watch mode.

When we get to the cache key, the entire config serialized as a string is part of the cache name, and that will change for every single invocation of Jest:

https://github.com/facebook/jest/blob/31b81ba37eb779161b6db78aa862c0bcd6465bd5/packages/jest-runtime/src/ScriptTransformer.js#L84

SimenB · 2019-01-29T15:45:05Z

We should probably revert a370c21 to make it stable

/cc @thymikee

SimenB · 2019-01-29T15:46:13Z

workaround is to set name in your config

SimenB · 2019-01-29T16:08:58Z

PR: #7746

viknes-interana · 2019-01-30T00:39:55Z

workaround is to set name in your config

Sorry, where do we set the name? I only have a single config and it is declared in package.json. Adding a name property inside the jest object didn't help.

csvan · 2019-01-30T08:00:40Z

Also wondering where to set the name option - it is not documented on https://jestjs.io/docs/en/configuration.html

Setting it in my jest.config.js did not fix the issue.

DylanVann · 2019-02-04T23:00:56Z

Setting name in my jest.config.js file does significantly speed up runs after the first one, but even after that I've found Jest 24 is about 50% slower than Jest 23.

Jest 23: 70s
Jest 24: 105s

Both of those times are from a warm start (I ran Jest twice before recording the time).

Unfortunately I can't share the code. It's running around 300 test suites, around 3000 tests, and around 300 snapshots.

Edit:

I also have api.cache.using(() => process.env.NODE_ENV) in my babel.config.js, doesn't seem to help.

joelgetaction · 2019-02-04T23:44:51Z

Any chance of getting a 24.0.1 patch release to fix this? :-) As it stands we are unable to use 24.0.0 due to the significant perf drop ... Thanks!

…

On Feb 4, 2019 at 3:01 PM, <Dylan Vann ***@***.***)> wrote: Setting name in my jest.config.js file does significantly speed up runs after the first one, but even after that I've found Jest 24 is about 50% slower than Jest 23. Jest 23: 70s Jest 24: 105s Both of those times are from a warm start (I ran Jest twice before recording the time). Unfortunately I can't share the code. It's running around 300 test suites, around 3000 tests, and around 300 snapshots. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub (#7732 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/AmWxt93AFetN_h5zaBoK1rQjV0g0VLAHks5vKLu4gaJpZM4aVLHW).

jeysal · 2019-02-05T15:12:49Z

24.1.0 is released

SimenB · 2019-02-05T15:19:10Z

Can people test that, and if they still have perf regressions compared to 23, open up a new issue?

joelgetaction · 2019-02-05T18:11:57Z

Will do and thanks!

…

On Feb 5, 2019 at 7:19 AM, <Simen Bekkhus ***@***.***)> wrote: Can people test that, and if they still have perf regressions compared to 23, open up a new issue? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub (#7732 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/AmWxtwS4-_2TkU66cQtzJyLwNO6yM_miks5vKaD8gaJpZM4aVLHW).

Jony-Y · 2019-03-10T14:21:09Z

using jest 24.3.1 vs jest 23.6.0

jest 24.3.1: 108s
jest 23.6.0: 66s

csvan · 2019-03-14T16:06:39Z

@Jony-Y that is not related to this bug though, see #7811

mucsi96 · 2019-08-01T12:36:53Z

Let me share my profiling here.
Specs:

Windows 10 Pro
Node 10.15.3
Intel Core i7-4702MQ 2.2GHz
8GB RAM
x64
SSD

Steps:

npx create-react-app my-app --typescript
change App.test.tsx
run npm test

CPU Profile:

CPU-20190801T141211.zip

Is it expected that 15 seconds are spent only with requring modules for single trivial React component and test?

Can someone with more experience on CPU profiling take a look?

jeysal · 2019-08-01T12:41:16Z

@mucsi96 a big part of that is likely micromatch, which is already known as a performance problem and will be fixed in 25

mucsi96 · 2019-08-01T13:16:38Z

Which Jest package depends on micromatch 3 causing the performance problem? Can I try bumping it within the package?

jeysal · 2019-08-01T13:21:14Z

more info in
#7963
#8032

mucsi96 · 2019-08-01T14:12:52Z

Thanks! I have tried to replace micromatch 3 with latest 4.0.2 in node_modules and achieved 4 seconds shorter test execution time. Initial time was 20 seconds on npx create-react-app my-app --typescript. So that`s 20% speed improvement. But I still feel it very slow. 16 seconds for single trivial React component and test feels too much for me.

Specs:

Windows 10 Pro
Node 10.15.3
Intel Core i7-4702MQ 2.2GHz
8GB RAM
x64
SSD

Steps:

npx create-react-app my-app --typescript
change App.test.tsx
run npm test

CPU Profile:

CPU-micromatch-4.zip

thymikee · 2019-08-01T14:24:15Z

@mucsi96 The issue with TS being slow may be caused by the order of module extension resolution. By default, Jest searches first for ['js', 'json', 'jsx', 'ts', 'tsx', 'node'] – so TS is supported, but at a performance cost. You can adjust the moduleFileExtensions config, to have ts and tsx at the very beginning, e.g. ['ts', 'tsx', 'js', 'json', 'jsx', 'node'].

However, looking at the source, CRA has its own list, which is not allowed to be overridden: https://github.com/facebook/create-react-app/blob/b96ac6c2c007db85812a6e9c0ec54509019c0fea/packages/react-scripts/config/paths.js#L49-L61. You could try passing it as a flag to the CLI, but not sure about that. I'd try adjusting the list in node_modules, making sure it works faster and then send a PR to CRA with adjusted moduleFileExtensions for TS

mucsi96 · 2019-08-01T14:27:51Z

Thanks for noting this. I have already submitted an different issue due to roots not set. facebook/create-react-app#7458

mucsi96 · 2019-08-01T14:47:38Z

I have tried ['ts', 'tsx', 'js', 'json', 'jsx', 'node'] but it made the execution even slower. Probably due to single React component and test vs. much more build-in modules.

mucsi96 · 2019-08-01T15:02:40Z

Also this issue seems to be not related to TypeScript. With npx create-react-app my-app the test execution is about 1 second faster then the TS version. So 18.69 seconds on my machine after adding roots to createJestConfig

Darth-koder007 · 2019-08-27T06:49:20Z

My project was using the old one jest-dom/extend-expect, so I replaced it with @testing-library/jest-dom/extend-expect and upgraded jest to 24.9.0, now tests and coverage are lightning fast.

github-actions · 2021-05-11T19:09:01Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

joelgetaction added the 🐛 Bug label Jan 28, 2019

SimenB added Needs Repro 💥 Regression and removed 🐛 Bug labels Jan 28, 2019

SimenB added Confirmed Hi-Pri and removed Needs Repro labels Jan 29, 2019

SimenB mentioned this issue Jan 29, 2019

fix: make sure generated name in config is stable across runs of Jest #7746

Merged

SimenB closed this as completed in #7746 Jan 29, 2019

milesj mentioned this issue Jan 29, 2019

Jest much slower in Travis CI than locally #7647

Closed

mkreiser mentioned this issue Feb 3, 2019

Jest devDependency is out of date (v23.6 --> v24) mkreiser/ESPN-Fantasy-Football-API#92

Closed

vitalibozhko mentioned this issue Feb 5, 2019

Jest 24 is slower than Jest 23.6.0 on the same test suite #7811

Closed

ravinderk mentioned this issue Mar 29, 2019

fix: increase jest test timeout to prevent frequent test failure impress-org/givewp#4060

Closed

1 task

schuchard mentioned this issue Aug 31, 2020

Introduce *IfNecessary Load Actions briebug/ngrx-auto-entity#148

Merged

github-actions bot locked as resolved and limited conversation to collaborators May 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jest 24 is 67% slower than Jest 23.6.0 on the same test suite #7732

Jest 24 is 67% slower than Jest 23.6.0 on the same test suite #7732

joelgetaction commented Jan 28, 2019

SimenB commented Jan 28, 2019

CrOrc commented Jan 28, 2019

SimenB commented Jan 28, 2019 •

edited

Loading

CrOrc commented Jan 28, 2019

lirbank commented Jan 28, 2019

overlookmotel commented Jan 29, 2019 •

edited

Loading

csvan commented Jan 29, 2019 •

edited

Loading

SimenB commented Jan 29, 2019 •

edited

Loading

SimenB commented Jan 29, 2019

SimenB commented Jan 29, 2019

SimenB commented Jan 29, 2019

viknes-interana commented Jan 30, 2019

csvan commented Jan 30, 2019

DylanVann commented Feb 4, 2019 •

edited

Loading

joelgetaction commented Feb 4, 2019 via email

jeysal commented Feb 5, 2019

SimenB commented Feb 5, 2019

joelgetaction commented Feb 5, 2019 via email

Jony-Y commented Mar 10, 2019

csvan commented Mar 14, 2019

mucsi96 commented Aug 1, 2019

jeysal commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

jeysal commented Aug 1, 2019

mucsi96 commented Aug 1, 2019 •

edited

Loading

thymikee commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

Darth-koder007 commented Aug 27, 2019

github-actions bot commented May 11, 2021

Jest 24 is 67% slower than Jest 23.6.0 on the same test suite #7732

Jest 24 is 67% slower than Jest 23.6.0 on the same test suite #7732

Comments

joelgetaction commented Jan 28, 2019

🐛 Bug Report

To Reproduce

Expected behavior

Link to repl or repo (highly encouraged)

Run npx envinfo --preset jest

SimenB commented Jan 28, 2019

CrOrc commented Jan 28, 2019

SimenB commented Jan 28, 2019 • edited Loading

CrOrc commented Jan 28, 2019

lirbank commented Jan 28, 2019

overlookmotel commented Jan 29, 2019 • edited Loading

csvan commented Jan 29, 2019 • edited Loading

SimenB commented Jan 29, 2019 • edited Loading

SimenB commented Jan 29, 2019

SimenB commented Jan 29, 2019

SimenB commented Jan 29, 2019

viknes-interana commented Jan 30, 2019

csvan commented Jan 30, 2019

DylanVann commented Feb 4, 2019 • edited Loading

joelgetaction commented Feb 4, 2019 via email

jeysal commented Feb 5, 2019

SimenB commented Feb 5, 2019

joelgetaction commented Feb 5, 2019 via email

Jony-Y commented Mar 10, 2019

csvan commented Mar 14, 2019

mucsi96 commented Aug 1, 2019

jeysal commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

jeysal commented Aug 1, 2019

mucsi96 commented Aug 1, 2019 • edited Loading

thymikee commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

mucsi96 commented Aug 1, 2019

Darth-koder007 commented Aug 27, 2019

github-actions bot commented May 11, 2021

Run `npx envinfo --preset jest`

SimenB commented Jan 28, 2019 •

edited

Loading

overlookmotel commented Jan 29, 2019 •

edited

Loading

csvan commented Jan 29, 2019 •

edited

Loading

SimenB commented Jan 29, 2019 •

edited

Loading

DylanVann commented Feb 4, 2019 •

edited

Loading

mucsi96 commented Aug 1, 2019 •

edited

Loading