[Bug]: Memory consumption issues on Node JS 16.11.0+

[EternallLight]

🚨 As of Node 21.1.0, this has been fixed upstream. Hopefully the fixes will be backported to v18 and v20 as well (as of writing (Oct. 26 2023) they have not. Node 18.20.0 and Node 20.10.0 have the backports), but that is up to the Node.js project and nothing we control from here. Note that (native) ESM still has memory leaks - that can be tracked here: #14605. If you're unable to upgrade your version of Node, you can use --workerIdleMemoryLimit in Jest 29 and later. See https://jestjs.io/docs/configuration/#workeridlememorylimit-numberstring 🚨

Version

27.0.6

Steps to reproduce

1. Install the latest Node JS (16.11.0 or later) or use the appropriate Docker image
2. Set up a project with a multiplicity Jest tests
3. Run node --expose-gc node_modules/.bin/jest --logHeapUsage and see how the memory consumption starts increasing.

Expected behavior

Since Jest calls global.gc() when Garbage Collector is exposed and --logHeapUsage flag is present, the memory usage should be stable.

Actual behavior

The memory usage increases with every new test

Additional context

We had some issues with Jest workers consuming all available RAM both on CI machine and locally.
After doing some research, we found that if we run Jest like the following node --expose-gc node_modules/.bin/jest --logHeapUsage, the heap size remains stable. After upgrading to Node JS v16.11.0, the issue was back. Node v16.10.0 works fine. I believe it was something accidentally introduced in the new Node, but it might be useful to take a look at this from Jest perspective in search of possible workarounds.
I'm also having the same behavior on my working machine, environment of which I'm pasting below 👇🏻

Environment

System:
    OS: macOS 11.6
    CPU: (8) x64 Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
  Binaries:
    Node: 16.11.0 - ~/.nvm/versions/node/v16.11.0/bin/node
    Yarn: 1.22.0 - ~/SomeFancyDir/webapp/node_modules/.bin/yarn
    npm: 8.0.0 - ~/.nvm/versions/node/v16.11.0/bin/npm
  npmPackages:
    jest: 27.0.6 => 27.0.6

[rthreei]

We're also experiencing this issue. Node 16.11+ and Jest v27 consumes significantly more memory. Node 16.10 and Jest v27 seems OK.

[blimmer]

We're also unable to update to the LTS version of Node 16 (at the time of writing 16.13.0) because of this issue. We bisected the changes and identified that the upgrade from Node 16.10 to 16.11 caused our large Jest suite to hang indefinitely.

I took a look at the Node 16.11 changelog and I think the most likely culprit for this issue comes from the V8 update to 9.4 (PR). In V8 9.4, the new Sparkplug compiler is enabled by default (see also this Node issue).

I was hoping I could try disabling sparkplug to see verify that this is the issue. node exposes a V8 option to disable (--no-sparkplug), but I don't think it's passing through to the Jest workers when I call it like this:

node --no-sparkplug node_modules/.bin/jest

I also tried setting the V8 option in jest-environment-node here: https://github.com/facebook/jest/blob/42b020f2931ac04820521cc8037b7c430eb2fa2f/packages/jest-environment-node/src/index.ts#L109 via

setFlagsFromString('--no-sparkplug');

but I didn't see any change. I'm not sure if that means Sparkplug isn't causing the problem or if I'm not setting the V8 flag properly in the jest workers.

@SimenB - I see you've committed a good deal to jest-environment-node - any tips for how I might pass that V8 flag down through all the workers? If it's possible (even via patch-pacakge or something) I'd be happy to give it a shot on our test suite that's exhibiting the problem.

So I'm not exactly positive that this is the cause of the issue, but it seems like a potentially promising place to start looking.

[SimenB]

any tips for how I might pass that V8 flag down through all the workers?

I would have thought https://github.com/facebook/jest/blob/e0b33b74b5afd738edc183858b5c34053cfc26dd/packages/jest-worker/src/workers/ChildProcessWorker.ts#L93-L94 made it so it was passed down...

[blimmer]

It's possible the sparkplug thing is a red herring. Nothing else jumped out at me from that changelog.

I captured some heap size metrics to show the scale of the difference.

Node 16.10.0 (baseline)

> node --expose-gc ./node_modules/.bin/jest --logHeapUsage --maxWorkers ${TEST_PARALLELISM:-8} "--reporters=default" "--reporters=jest-junit"
(14.297 s, 211 MB heap size)
(16.811 s, 211 MB heap size)
(17.913 s, 211 MB heap size)
(18.382 s, 213 MB heap size)
(18.38 s, 214 MB heap size)
(19.122 s, 238 MB heap size)
(19.241 s, 211 MB heap size)
(21.969 s, 222 MB heap size)
(10.783 s, 268 MB heap size)
(6.974 s, 268 MB heap size)
(7.575 s, 267 MB heap size)
(7.404 s, 269 MB heap size)
(6.788 s, 288 MB heap size)
(6.907 s, 267 MB heap size)
(10.324 s, 268 MB heap size)
(6.796 s, 276 MB heap size)
(5.177 s, 282 MB heap size)
(5.638 s, 269 MB heap size)
(5.702 s, 269 MB heap size)
(268 MB heap size)
(6.432 s, 271 MB heap size)
(7.227 s, 268 MB heap size)
(7.276 s, 272 MB heap size)
(5.192 s, 278 MB heap size)
(271 MB heap size)
(5.053 s, 286 MB heap size)
(269 MB heap size)
(5.3 s, 272 MB heap size)
(281 MB heap size)
(274 MB heap size)
(281 MB heap size)
(5.018 s, 272 MB heap size)
(274 MB heap size)
(288 MB heap size)
(275 MB heap size)
(272 MB heap size)
(276 MB heap size)
(278 MB heap size)
(285 MB heap size)
(274 MB heap size)
(290 MB heap size)
(277 MB heap size)
(275 MB heap size)
(278 MB heap size)
(278 MB heap size)
(280 MB heap size)
(287 MB heap size)
(277 MB heap size)
(293 MB heap size)
(280 MB heap size)
(277 MB heap size)
(281 MB heap size)
(281 MB heap size)
(280 MB heap size)
(283 MB heap size)
(289 MB heap size)
(296 MB heap size)
(283 MB heap size)
(280 MB heap size)
(284 MB heap size)
(283 MB heap size)
(282 MB heap size)
(285 MB heap size)
(292 MB heap size)
(298 MB heap size)
(282 MB heap size)
(285 MB heap size)
(286 MB heap size)
(286 MB heap size)
(285 MB heap size)
(287 MB heap size)
(295 MB heap size)
(301 MB heap size)
(285 MB heap size)
(288 MB heap size)
(289 MB heap size)
(289 MB heap size)
(288 MB heap size)
(290 MB heap size)
(298 MB heap size)
(216 MB heap size)
(304 MB heap size)
(288 MB heap size)
(290 MB heap size)
(292 MB heap size)
(292 MB heap size)
(296 MB heap size)
(294 MB heap size)
(295 MB heap size)
(293 MB heap size)
(6.766 s, 283 MB heap size)
(7.887 s, 273 MB heap size)
(7.654 s, 308 MB heap size)
(8.406 s, 272 MB heap size)
(298 MB heap size)
(298 MB heap size)
(297 MB heap size)
(295 MB heap size)
(297 MB heap size)
(300 MB heap size)
(289 MB heap size)
(305 MB heap size)
(301 MB heap size)
(300 MB heap size)
(291 MB heap size)
(297 MB heap size)
(300 MB heap size)
(303 MB heap size)
(303 MB heap size)
(292 MB heap size)
(303 MB heap size)
(308 MB heap size)
(311 MB heap size)
(300 MB heap size)
(303 MB heap size)
(306 MB heap size)
(295 MB heap size)
(306 MB heap size)
(310 MB heap size)
(305 MB heap size)
(297 MB heap size)
(303 MB heap size)
(306 MB heap size)
(309 MB heap size)
(297 MB heap size)
(313 MB heap size)
(310 MB heap size)
(307 MB heap size)
(305 MB heap size)
(299 MB heap size)
(308 MB heap size)
(311 MB heap size)
(315 MB heap size)
(300 MB heap size)
(311 MB heap size)
(220 MB heap size)
(208 MB heap size)
(310 MB heap size)
(302 MB heap size)
(309 MB heap size)
(242 MB heap size)
(311 MB heap size)
(284 MB heap size)
(322 MB heap size)
(296 MB heap size)
(312 MB heap size)
(313 MB heap size)
(315 MB heap size)
(317 MB heap size)
(316 MB heap size)
(304 MB heap size)
(315 MB heap size)
(308 MB heap size)
(315 MB heap size)
(317 MB heap size)
(318 MB heap size)
(317 MB heap size)
(320 MB heap size)
(306 MB heap size)
(311 MB heap size)
(317 MB heap size)
(320 MB heap size)
(318 MB heap size)
(320 MB heap size)
(323 MB heap size)
(320 MB heap size)
(310 MB heap size)
(313 MB heap size)
(323 MB heap size)
(320 MB heap size)
(321 MB heap size)
(340 MB heap size)
(326 MB heap size)
(323 MB heap size)
(312 MB heap size)
(326 MB heap size)
(316 MB heap size)
(322 MB heap size)
(328 MB heap size)
(323 MB heap size)
(326 MB heap size)
(343 MB heap size)
(315 MB heap size)
(329 MB heap size)
(325 MB heap size)
(319 MB heap size)
(328 MB heap size)
(326 MB heap size)
(331 MB heap size)
(345 MB heap size)
(318 MB heap size)
(331 MB heap size)
(328 MB heap size)
(321 MB heap size)
(329 MB heap size)
(332 MB heap size)
(334 MB heap size)
(348 MB heap size)
(320 MB heap size)
(334 MB heap size)
(331 MB heap size)
(324 MB heap size)
(335 MB heap size)
(332 MB heap size)
(337 MB heap size)
(350 MB heap size)
(323 MB heap size)
(265 MB heap size)
(333 MB heap size)
(337 MB heap size)
(302 MB heap size)
(240 MB heap size)
(327 MB heap size)
(257 MB heap size)
(266 MB heap size)
(334 MB heap size)
(337 MB heap size)
(9.083 s, 342 MB heap size)
(9.235 s, 319 MB heap size)
(9.967 s, 321 MB heap size)
(9.455 s, 345 MB heap size)
(9.686 s, 359 MB heap size)
(12.938 s, 313 MB heap size)
(13.969 s, 340 MB heap size)
(14.6 s, 333 MB heap size)
(6.149 s, 340 MB heap size)
(8.209 s, 347 MB heap size)
(9.345 s, 337 MB heap size)
(10 s, 344 MB heap size)
(8.202 s, 360 MB heap size)
(7.368 s, 333 MB heap size)
(6.233 s, 342 MB heap size)
(6.243 s, 359 MB heap size)
(6.108 s, 342 MB heap size)
(363 MB heap size)
(5.324 s, 315 MB heap size)
(332 MB heap size)
(6.147 s, 338 MB heap size)
(345 MB heap size)
(5.909 s, 341 MB heap size)
(7.36 s, 325 MB heap size)
(5.155 s, 342 MB heap size)
(324 MB heap size)
(5.058 s, 364 MB heap size)
(313 MB heap size)
(5.34 s, 276 MB heap size)
(273 MB heap size)
(342 MB heap size)
(281 MB heap size)
(271 MB heap size)
(327 MB heap size)
(273 MB heap size)
(344 MB heap size)
(276 MB heap size)
(296 MB heap size)
(273 MB heap size)
(281 MB heap size)
(271 MB heap size)
(276 MB heap size)
(289 MB heap size)
(299 MB heap size)
(347 MB heap size)
(277 MB heap size)
(275 MB heap size)
(283 MB heap size)
(274 MB heap size)
(279 MB heap size)
(291 MB heap size)
(350 MB heap size)
(302 MB heap size)
(279 MB heap size)
(278 MB heap size)
(286 MB heap size)
(277 MB heap size)
(281 MB heap size)
(294 MB heap size)
(353 MB heap size)
(283 MB heap size)
(284 MB heap size)
(289 MB heap size)
(281 MB heap size)
(281 MB heap size)
(285 MB heap size)
(297 MB heap size)
(285 MB heap size)
(355 MB heap size)
(287 MB heap size)
(283 MB heap size)
(292 MB heap size)
(282 MB heap size)
(287 MB heap size)
(300 MB heap size)
(286 MB heap size)
(358 MB heap size)
(287 MB heap size)
(294 MB heap size)
(290 MB heap size)
(285 MB heap size)
(290 MB heap size)
(302 MB heap size)
(289 MB heap size)
(297 MB heap size)
(290 MB heap size)
(292 MB heap size)
(361 MB heap size)
(365 MB heap size)
(5.125 s, 298 MB heap size)
(6.13 s, 295 MB heap size)
(6.526 s, 304 MB heap size)
(6.928 s, 301 MB heap size)
(7.982 s, 281 MB heap size)
(8.516 s, 317 MB heap size)
(9.841 s, 313 MB heap size)
(5.477 s, 295 MB heap size)
(5.506 s, 312 MB heap size)
(296 MB heap size)
(294 MB heap size)
(301 MB heap size)
(298 MB heap size)
(326 MB heap size)
(301 MB heap size)
(297 MB heap size)
(294 MB heap size)
(294 MB heap size)
(330 MB heap size)
(295 MB heap size)
(301 MB heap size)
(330 MB heap size)
(296 MB heap size)
(323 MB heap size)
(297 MB heap size)
(297 MB heap size)
(316 MB heap size)
(297 MB heap size)
(304 MB heap size)
(332 MB heap size)
(298 MB heap size)
(308 MB heap size)
(319 MB heap size)
(300 MB heap size)
(300 MB heap size)
(300 MB heap size)
(334 MB heap size)
(306 MB heap size)
(301 MB heap size)
(310 MB heap size)
(303 MB heap size)
(322 MB heap size)
(302 MB heap size)
(302 MB heap size)
(337 MB heap size)
(310 MB heap size)
(212 MB heap size)
(209 MB heap size)
(304 MB heap size)
(253 MB heap size)
(230 MB heap size)
(265 MB heap size)
(286 MB heap size)
(312 MB heap size)
(5.316 s, 317 MB heap size)
(5.007 s, 307 MB heap size)
(8.481 s, 309 MB heap size)
(8.669 s, 289 MB heap size)
(10.421 s, 313 MB heap size)
(283 MB heap size)
(5.801 s, 316 MB heap size)
(336 MB heap size)
(13.269 s, 344 MB heap size)
(288 MB heap size)
(14.837 s, 338 MB heap size)
(288 MB heap size)
(331 MB heap size)
(285 MB heap size)
(302 MB heap size)
(287 MB heap size)
(340 MB heap size)
(297 MB heap size)
(297 MB heap size)
(287 MB heap size)
(313 MB heap size)
(291 MB heap size)
(343 MB heap size)
(300 MB heap size)
(279 MB heap size)
(298 MB heap size)
(290 MB heap size)
(293 MB heap size)
(314 MB heap size)
(303 MB heap size)
(345 MB heap size)
(282 MB heap size)
(301 MB heap size)
(25.518 s, 290 MB heap size)
(293 MB heap size)
(295 MB heap size)
(317 MB heap size)
(305 MB heap size)
(284 MB heap size)
(348 MB heap size)
(303 MB heap size)
(295 MB heap size)
(298 MB heap size)
(320 MB heap size)
(306 MB heap size)
(308 MB heap size)
(287 MB heap size)
(351 MB heap size)
(306 MB heap size)
(298 MB heap size)
(301 MB heap size)
(322 MB heap size)
(310 MB heap size)
(309 MB heap size)
(290 MB heap size)
(353 MB heap size)
(308 MB heap size)
(300 MB heap size)
(264 MB heap size)
(303 MB heap size)
(312 MB heap size)
(326 MB heap size)
(292 MB heap size)
(313 MB heap size)
(268 MB heap size)
(356 MB heap size)
(233 MB heap size)
(220 MB heap size)
(254 MB heap size)
(252 MB heap size)
(232 MB heap size)
(255 MB heap size)
(232 MB heap size)

Node 16.11.0 (problem introduced)

> node --expose-gc --no-sparkplug ./node_modules/.bin/jest --logHeapUsage --maxWorkers ${TEST_PARALLELISM:-8} "--reporters=default" "--reporters=jest-junit"
(41.727 s, 217 MB heap size)
(42.441 s, 217 MB heap size)
(42.452 s, 216 MB heap size)
(45.134 s, 217 MB heap size)
(45.868 s, 220 MB heap size)
(46.713 s, 245 MB heap size)
(49.371 s, 247 MB heap size)
(9.725 s, 328 MB heap size)
(12.574 s, 329 MB heap size)
(54.825 s, 218 MB heap size)
(8.787 s, 340 MB heap size)
(13.189 s, 329 MB heap size)
(13.471 s, 328 MB heap size)
(10.219 s, 341 MB heap size)
(14.752 s, 333 MB heap size)
(8.109 s, 397 MB heap size)
(10.887 s, 386 MB heap size)
(13.786 s, 387 MB heap size)
(8.03 s, 385 MB heap size)
(13.016 s, 388 MB heap size)
(9.036 s, 419 MB heap size)
(332 MB heap size)
(16.237 s, 328 MB heap size)
(10.056 s, 390 MB heap size)
(7.183 s, 458 MB heap size)
(6.975 s, 446 MB heap size)
(9.844 s, 411 MB heap size)
(7.222 s, 423 MB heap size)
(7.276 s, 426 MB heap size)
(6.49 s, 520 MB heap size)
(7.583 s, 444 MB heap size)
(14.041 s, 416 MB heap size)
(10.527 s, 386 MB heap size)
(6.191 s, 415 MB heap size)
(8.422 s, 510 MB heap size)
(6.528 s, 581 MB heap size)
(7.28 s, 486 MB heap size)
(9.558 s, 523 MB heap size)
(7.271 s, 423 MB heap size)
(6.011 s, 545 MB heap size)
(7.171 s, 477 MB heap size)
(9.289 s, 414 MB heap size)
(16.893 s, 496 MB heap size)
(9.017 s, 642 MB heap size)
(7.173 s, 521 MB heap size)
(9.823 s, 548 MB heap size)
(6.973 s, 606 MB heap size)
(7.009 s, 565 MB heap size)
(6.953 s, 416 MB heap size)
(8.185 s, 483 MB heap size)
(581 MB heap size)
(8.146 s, 489 MB heap size)
(5.061 s, 609 MB heap size)
(5.397 s, 666 MB heap size)
(6.634 s, 541 MB heap size)
(5.866 s, 476 MB heap size)
(8.275 s, 670 MB heap size)
(529 MB heap size)
(9.576 s, 478 MB heap size)
(6.206 s, 643 MB heap size)
(7.119 s, 670 MB heap size)
(6.208 s, 538 MB heap size)
(10.041 s, 550 MB heap size)
(6.383 s, 623 MB heap size)
(8.896 s, 727 MB heap size)
(6.355 s, 477 MB heap size)
(6.294 s, 704 MB heap size)
(5.112 s, 599 MB heap size)
(12.077 s, 748 MB heap size)
(5.699 s, 731 MB heap size)
(730 MB heap size)
(6.233 s, 722 MB heap size)
(7.113 s, 549 MB heap size)
(8.579 s, 684 MB heap size)
(6.936 s, 776 MB heap size)
(6.933 s, 659 MB heap size)
(6.843 s, 794 MB heap size)
(10.345 s, 539 MB heap size)
(5.048 s, 842 MB heap size)
(609 MB heap size)
(5.361 s, 783 MB heap size)
(6.001 s, 682 MB heap size)
(6.456 s, 828 MB heap size)
(5.785 s, 854 MB heap size)
(5.447 s, 451 MB heap size)
(5.668 s, 670 MB heap size)
(5.305 s, 843 MB heap size)
(7.549 s, 720 MB heap size)
(6.11 s, 903 MB heap size)
(5.761 s, 743 MB heap size)
(731 MB heap size)
(6.164 s, 889 MB heap size)
(5.513 s, 914 MB heap size)
(5.31 s, 904 MB heap size)
(5.111 s, 963 MB heap size)
(6.897 s, 512 MB heap size)
(6.214 s, 782 MB heap size)
(5.568 s, 805 MB heap size)
(573 MB heap size)
(5.445 s, 949 MB heap size)
(1024 MB heap size)
(6.722 s, 793 MB heap size)
(842 MB heap size)
(6.405 s, 964 MB heap size)
(8.361 s, 975 MB heap size)
(5.613 s, 865 MB heap size)
(902 MB heap size)
(5.719 s, 1009 MB heap size)
(5.861 s, 1084 MB heap size)
(5.734 s, 854 MB heap size)
(5.225 s, 1026 MB heap size)
(7.716 s, 633 MB heap size)
(5.191 s, 926 MB heap size)
(6.251 s, 974 MB heap size)
(1069 MB heap size)
(1085 MB heap size)
(1145 MB heap size)
(6.744 s, 962 MB heap size)
(5.617 s, 915 MB heap size)
(6.088 s, 653 MB heap size)
(1034 MB heap size)
(5.324 s, 987 MB heap size)
(5.146 s, 1130 MB heap size)
(1146 MB heap size)
(713 MB heap size)
(5.612 s, 1223 MB heap size)
(5.552 s, 975 MB heap size)
(1094 MB heap size)
(6.886 s, 963 MB heap size)
(6.49 s, 1048 MB heap size)
(1205 MB heap size)
(5.253 s, 1192 MB heap size)
(5.241 s, 775 MB heap size)
(1265 MB heap size)
(1156 MB heap size)
(1022 MB heap size)
(6.167 s, 1035 MB heap size)
(5.235 s, 1266 MB heap size)
(5.017 s, 1273 MB heap size)
(7.344 s, 1107 MB heap size)
(5.872 s, 1329 MB heap size)
(5.267 s, 1084 MB heap size)
(6.312 s, 1216 MB heap size)
(9.271 s, 837 MB heap size)
(1329 MB heap size)
(1315 MB heap size)
(10.02 s, 1107 MB heap size)
(1389 MB heap size)
(6.675 s, 983 MB heap size)
(1144 MB heap size)
(1276 MB heap size)
(5.482 s, 775 MB heap size)
(1375 MB heap size)
(5.25 s, 1390 MB heap size)
(5.264 s, 1204 MB heap size)
(5.69 s, 1449 MB heap size)
(5.953 s, 1042 MB heap size)
(6.997 s, 1031 MB heap size)
(1339 MB heap size)
(6.208 s, 845 MB heap size)
(5.602 s, 1435 MB heap size)
(1265 MB heap size)
(1092 MB heap size)
(6.002 s, 1103 MB heap size)
(898 MB heap size)
(9.446 s, 1462 MB heap size)
(6.544 s, 1510 MB heap size)
(6.159 s, 1400 MB heap size)
(1495 MB heap size)
(1328 MB heap size)
(5.141 s, 1152 MB heap size)
(1513 MB heap size)
(5.133 s, 957 MB heap size)
(5.965 s, 1164 MB heap size)
(1571 MB heap size)
(6.002 s, 1397 MB heap size)
(5.063 s, 1556 MB heap size)
(5.68 s, 1388 MB heap size)
(1018 MB heap size)
(6.518 s, 1212 MB heap size)
(5.141 s, 1631 MB heap size)
(6.407 s, 1574 MB heap size)
(6.51 s, 1224 MB heap size)
(5.082 s, 1466 MB heap size)
(1616 MB heap size)
(1078 MB heap size)
(1448 MB heap size)
(6.306 s, 1273 MB heap size)
(5.142 s, 1691 MB heap size)
(1677 MB heap size)
(5.811 s, 1077 MB heap size)
(1139 MB heap size)
(6.226 s, 1435 MB heap size)
(1508 MB heap size)
(1138 MB heap size)
(6.122 s, 1213 MB heap size)
(1736 MB heap size)
(1199 MB heap size)
(1496 MB heap size)
(15.376 s, 1528 MB heap size)
(1272 MB heap size)
(5.026 s, 1198 MB heap size)
(10.881 s, 1763 MB heap size)
(1258 MB heap size)
(8.77 s, 1580 MB heap size)
(1556 MB heap size)
(5.052 s, 1797 MB heap size)
(1586 MB heap size)
(1333 MB heap size)
(1322 MB heap size)
(1631 MB heap size)
(5.567 s, 1259 MB heap size)
(1616 MB heap size)
(5.411 s, 1857 MB heap size)
(6.565 s, 1755 MB heap size)
(1646 MB heap size)
(1691 MB heap size)
(1322 MB heap size)
(5.522 s, 1382 MB heap size)
(5.783 s, 1393 MB heap size)
(1549 MB heap size)
(5.275 s, 1676 MB heap size)
(1918 MB heap size)
(1442 MB heap size)
(1751 MB heap size)
(5.368 s, 1382 MB heap size)
(5.263 s, 1454 MB heap size)
(1661 MB heap size)
(5.433 s, 1736 MB heap size)
(1978 MB heap size)
(1502 MB heap size)
(1442 MB heap size)
(1812 MB heap size)
(1514 MB heap size)
(1737 MB heap size)
(1796 MB heap size)
(2038 MB heap size)
(17.555 s, 1800 MB heap size)
(1563 MB heap size)
(1502 MB heap size)
(1872 MB heap size)
(1574 MB heap size)
(1797 MB heap size)
(1428 MB heap size)
(1857 MB heap size)
(2099 MB heap size)
(1750 MB heap size)
(1562 MB heap size)
(6.206 s, 1569 MB heap size)
(1635 MB heap size)
(1933 MB heap size)
(6.031 s, 1560 MB heap size)
(1917 MB heap size)
(1862 MB heap size)
(2159 MB heap size)
(1623 MB heap size)
(1629 MB heap size)
(1694 MB heap size)
(5.348 s, 1637 MB heap size)
(1938 MB heap size)
(2218 MB heap size)
(6.073 s, 1978 MB heap size)
(1689 MB heap size)
(1683 MB heap size)
(1755 MB heap size)
(1543 MB heap size)
(1697 MB heap size)
(1998 MB heap size)
(2038 MB heap size)
(5.306 s, 2285 MB heap size)
(1749 MB heap size)
(1815 MB heap size)
(1677 MB heap size)
(2072 MB heap size)
(1757 MB heap size)
(2059 MB heap size)
(2098 MB heap size)
(2345 MB heap size)
(1875 MB heap size)
(1753 MB heap size)
(1817 MB heap size)
(2115 MB heap size)
(8.092 s, 1810 MB heap size)
(2118 MB heap size)
(2158 MB heap size)
(2405 MB heap size)
(1936 MB heap size)
(1814 MB heap size)
(5.142 s, 1877 MB heap size)
(1869 MB heap size)
(2138 MB heap size)
(5.164 s, 2192 MB heap size)
(2180 MB heap size)
(1874 MB heap size)
(2465 MB heap size)
(1996 MB heap size)
(1931 MB heap size)
(5.237 s, 1938 MB heap size)
(2252 MB heap size)
(5.079 s, 2247 MB heap size)
(2239 MB heap size)
(2057 MB heap size)
(1934 MB heap size)
(2525 MB heap size)
(1997 MB heap size)
(1990 MB heap size)
(2312 MB heap size)
(2307 MB heap size)
(2305 MB heap size)
(2116 MB heap size)
(2258 MB heap size)
(1995 MB heap size)
(2586 MB heap size)
(5.015 s, 2058 MB heap size)
(2050 MB heap size)
(2365 MB heap size)
(2373 MB heap size)
(2176 MB heap size)
(2390 MB heap size)
(2055 MB heap size)
(2337 MB heap size)
(2645 MB heap size)
(2118 MB heap size)
(2110 MB heap size)
(2433 MB heap size)
(2053 MB heap size)
(2237 MB heap size)
(2114 MB heap size)
(2450 MB heap size)
(2447 MB heap size)
(2178 MB heap size)
(2706 MB heap size)
(2180 MB heap size)
(2296 MB heap size)
(2175 MB heap size)
(2511 MB heap size)
(2507 MB heap size)
(2239 MB heap size)
(2767 MB heap size)
(2553 MB heap size)
(2241 MB heap size)
(2163 MB heap size)
(2235 MB heap size)
(2362 MB heap size)
(2567 MB heap size)
(5.929 s, 2598 MB heap size)
(2826 MB heap size)
(2613 MB heap size)
(2301 MB heap size)
(2293 MB heap size)
(2422 MB heap size)
(2301 MB heap size)
(2627 MB heap size)
(2284 MB heap size)
(2623 MB heap size)
(2887 MB heap size)
(2673 MB heap size)
(2286 MB heap size)
(2367 MB heap size)
(2369 MB heap size)
(2361 MB heap size)
(2687 MB heap size)
(2684 MB heap size)
(2419 MB heap size)
(2733 MB heap size)
(2963 MB heap size)
(2429 MB heap size)
(2427 MB heap size)
(2421 MB heap size)
(2747 MB heap size)
(2495 MB heap size)
(2489 MB heap size)
(2744 MB heap size)
(2793 MB heap size)
(2488 MB heap size)
(3007 MB heap size)
(2481 MB heap size)
(2807 MB heap size)
(2549 MB heap size)
(2804 MB heap size)
(2555 MB heap size)
(2854 MB heap size)
(2547 MB heap size)
(3067 MB heap size)
(2541 MB heap size)
(2867 MB heap size)
(2863 MB heap size)
(2615 MB heap size)
(2609 MB heap size)
(2607 MB heap size)
(2914 MB heap size)
(3127 MB heap size)
(2601 MB heap size)
(2927 MB heap size)
(2669 MB heap size)
(2667 MB heap size)
(2974 MB heap size)
(3187 MB heap size)
(2662 MB heap size)
(2681 MB heap size)
(2988 MB heap size)
(2729 MB heap size)
(2984 MB heap size)
(2727 MB heap size)
(2723 MB heap size)
(3247 MB heap size)
(2757 MB heap size)
(3048 MB heap size)
(2789 MB heap size)
(5.435 s, 3044 MB heap size)
(11.665 s, 2788 MB heap size)
(12.754 s, 2859 MB heap size)

I'll log out the code you posted to make sure it's being passed down.

[blimmer]

Yep, @SimenB you were correct - the --no-sparkplug flag does appear to be making it down to the workers. Thanks for pointing me to that code.

@EternallLight and @rthreei - is there anything about your codebase that's notable to potentially cause this issue? The only thing I can think of on my end is that we make heavy use of async_hooks and have historically had issues upgrading in the 16.x series related to them.

I'm sure the jest team would love to have some kind of reproducibility to try to look more closely at this, but I'm struggling to develop a small reproducible case.

[rthreei]

@blimmer nothing in particular is notable about our codebase. It's not a small codebase, but not very big either. Prior to 16.11, there were known memory leaks when running test suite (potentially around lodash); maybe made worse by 16.11.

[SimenB]

If it's specifically in 16.11, you can probably try to build node yourself and bisect https://github.com/nodejs/node/compare/v16.10.0..v16.11.0. Figuring out which commit that introduced it might help understand when one (or more) of your code, Node and Jest does wrong 🙂

[pustovalov]

Looks like in new version of node, garbage collector will no longer clear memory until it reaches the limit.

RAM: 4GB

Node v14.17.3

node --expose-gc --trace_gc ./node_modules/.bin/jest app/javascript --forceExit --ci -w=2 --coverage --logHeapUsage > log/jest_memory_usage.log

Max memory usage:

[552:0x49fcc00]   116300 ms: Scavenge 700.3 (726.1) -> 692.0 (729.6) MB, 4.4 / 0.0 ms  (average mu = 0.983, current mu = 0.990) allocation failure 

full log:
https://gist.github.com/pustovalov/ad8bcd84b0b6bd6abec301982e03ed55

Node v16.13.0

node --expose-gc --trace_gc ./node_modules/.bin/jest app/javascript --forceExit --ci -w=2 --coverage --logHeapUsage > log/current-node-16.log

Max memory usage:

[10321:0x7fdd67100000]   865787 ms: Scavenge 2939.4 (3351.0) -> 2924.0 (3351.0) MB, 3.1 / 0.0 ms  (average mu = 0.346, current mu = 0.315) allocation failure 

full log:
https://gist.github.com/pustovalov/01cc1f484566f4b692fb405335b5f78d

Node v16.13.0 with max-old-space-size

based on how much memory the process was using when it used node v14, set a limit in 700mb

node --expose-gc --trace_gc --max-old-space-size=700 ./node_modules/.bin/jest app/javascript --forceExit --ci -w=2 --coverage --logHeapUsage > log/jest_memory_usage.log

Max memory usage:

[1040:0x66e7250]   193826 ms: Scavenge 603.6 (640.0) -> 592.4 (643.3) MB, 9.2 / 0.0 ms  (average mu = 0.823, current mu = 0.844) allocation failure 

full log:
https://gist.github.com/pustovalov/a86545aa38708d6dda479ee9a1bc4e1d

If you set a limit that will be equal to the maximum amount of RAM, then the process will crash because the cleaning will begin only when this limit is reached

  System:
    OS: macOS 11.6
    CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  Binaries:
    Node: 16.13.0 - /usr/local/Cellar/node@16/16.13.0/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.1.0 - /usr/local/Cellar/node@16/16.13.0/bin/npm
  npmPackages:
    jest: ^27.3.1 => 27.3.1

[luiz290788]

If it's specifically in 16.11, you can probably try to build node yourself and bisect https://github.com/nodejs/node/compare/v16.10.0..v16.11.0. Figuring out which commit that introduced it might help understand when one (or more) of your code, Node and Jest does wrong 🙂

I've tried that. I was able to reproduce the issue on the last commit on v8 upgrade. It looks like the problem is related to v8 upgrade inside of node.

[B4nan]

Looks like I am hitting this as well. It happens only when doing coverage (as it eats more memory), and it ends up with hard failure. Locking node version to 16.10 helps, node 17 fails the same. Fails locally too.

Maybe one interesting note, it started to happen after I upgraded node-mongodb to v4, failed pipeline here.

[jgoz]

Trying to find potentially related issues in NodeJS and this one jumped out at me: nodejs/node#40014

And the associated V8 bug: https://bugs.chromium.org/p/v8/issues/detail?id=12198

[luiz290788]

I was trying to isolate the issue from our codebase and created this repo. It looks like that even for very simple test suites, the problem is there.

[mousetree]

We had consistent out of memory errors on node 16.13, reverting to 16.10 solved the problem for us.

[mb21]

Same here. On node 16.13.1, heap size goes up to 500MB while on node 16.10, it stays at 50MB with the following repro:

mkdir test
cd test
npm install jest
mkdir src
echo 'it("a", () => expect(0).toBeFalsy())' > src/a.test.js
for i in {1..100}; do cp src/a.test.js "src/a$i.test.js"; done
./node_modules/jest/bin/jest.js --runInBand --logHeapUsage

Edit: install Volta and do volta run --node 20.1.0 -- node ./node_modules/jest/bin/jest.js --runInBand --logHeapUsag to test a specific node version

[LukeNotable]

Running mb21's test on various Node versions confirms that everything after 16.10 (including 16.13.1 and 17.2.0) shows the same ever-growing heap, also running 3 times as slow.

We weren't able to get CI to finish running our tests under the latest Node 16, although somehow Node 17 was fine. 🤔

[lukeapage]

I've just tracked down this issue - for us it is a 2-5x performance regression (hopefully those key words help people find this). I've also tested the latest nodejs 16 & 17 versions without luck.

running a subset of tests, Memory in windows task manager on 16.10 reaches 900mb per worker. On 16.11.0 it reaches 1800mb per worker.

[tzimmermann]

Also hitting this after upgrading to Node 16.13. It's quite bad for us, as we cannot resolve some CVEs that have been fixed in 16.13 already.

I understand the underlying v8 issue is closed (wontfix), so I don't have much hope we'll have this fixed in a new node version any time soon.

So I am wondering:

• Are there plans to solve/work around this issue somehow from the jest side?
• Is anyone aware if this memory leak has implications beyond slowing down jest, i.e. should we hold back with updating node servers running in production?

[jgoz]

• Is anyone aware if this memory leak has implications beyond slowing down jest, i.e. should we hold back with updating node servers running in production?

I did some local profiling and didn't see any significant difference in memory consumption between 16.10 and 16.13 for our production app, but we aren't using vm.Script

[LukeNotable]

Following a suggestion on the Node issue, I tried running with the Node option --no-compilation-cache. Suddenly my CI with 5000 tests works again, and furthermore it finishes without the ~1GB of memory leakage that I always saw under Node 14 (and of course without the 5GB+ that would leak under Node 16.13 until OOM). The downside is that it seems to take about 25–50% longer to finish.

[amiedes]

We've been recently hit by this bug (or combination of them) in several services of our stack. It's been an inconvenience for use since once the heap size exceeded the 2GB available by GitHub Actions machines, our CI/CD process was failing 100% of the time.

I believe the "modules accumulating in the heap" bug has been present in Jest for some time, but that in combination with the changes in Node from v16.10 to v16.11 has made it much more severe. We're certain that bug exists in Jest, as we've managed to replicate it with a "dummy repo", as several other people have stated.

The fact that the ticket created in Node JS was closed as a WONTFIX is a bit worrying. I currently lack the knowledge to judge the discussions which happened there, but if that wasn't a bug at the end probably the solution falls on the Jest side.

After trying to extract as much information from the different GitHub issues that have been created in Jest, TS-Jest and Node, and trying several approaches, the only path for us was to downgrade back to Node v16.10.

These are some of the statistics we gathered in the process:

Node	@jest/core	isolatedModules	runInBand	Comp. cache	Initial H	Final H	H ratio	Time
17.2.0	27.3.1	true	true	true	133 MB	2236 MB	~33 MB	144 s
16.13.0	27.3.1	true	true	true	131 MB	2220 MB	~30 MB	130 s
16.13.0	27.3.1	true	true	false	126 MB	1712 MB	~25 MB	113 s
16.13.0	27.3.1	false	true	true	287 MB	2393 MB	~30 MB	140 s
16.10.0	27.3.1	true	true	true	123 MB	790 MB	~10 MB	77 s
16.10.0	27.3.1	true	true	false	118 MB	1676 MB	~25 MB	110 s
16.10.0	27.3.1	false	true	true	280 MB	952 MB	~10 MB	90 s

Some explanations about the columns:

• Node: version of NodeJS
• @jest/core: version of the Jest core
• isolatedModules: the setting defined via jest.config.json
• runInBand: specified by the --runInBand Jest flag
• Comp. cache: specified by the NodeJS --no-compilation-cache flag
• Initial H: heap size after running the first test
• Final H: heap size after running the last test
• H ratio: heap size increase after each test
• Time: time the whole suite needed to run

The most relevant tickets related to this matter:

• Module caching memory leak #10550
• [Bug]: Memory consumption issues on Node JS 16.11.0+ #11956
• [Bug]: --runInBand seems to leak memory when importing the same module multiple times #12142
• Memory Leak when running multiple test. Especially with --runInBand #7311
• Memory Leak on ridiculously simple repo #7874
• Module caching memory leak kulshekhar/ts-jest#1967
• vm Script memory leak in Node.js 14 / 16 nodejs/node#40014
• https://bugs.chromium.org/p/v8/issues/detail?id=12198

We hope this issue is given the importance it deserves, as keeping the Node version pinned to v16.10 or relying on bigger CI machines to compensate this is a poor strategy for the future.

[lukeapage]

@amiedes it looks to me like the v8 bug was closed as won’t fix but the nodejs bug is still open.

I’m just a interested spectator and don’t have a 100% understanding but it seems to me like no one has indicated a belief this is a jest bug.

[pustovalov]

@amiedes can you try to run jest in this way?

node --expose-gc --trace_gc --max-old-space-size=700 ./node_modules/.bin/jest app/javascript --runInBand --logHeapUsage > log/jest_memory_usage.log

[xereda]

Can confirm: 21.1.0 means we can finally run our massive jest -i database test suite again (previously we had to split it into 4 separate runs to avoid running out of RAM). Total execution time is down from 2m12 for the four split test suites to 1m33 for running the entire lot. Huge thanks to everyone involved, and we'd love to see this backported to Node 20 (and ideally Node 18) so we don't have to split the runs in CI any more. Thanks again ❤️

What is the "-i" parameter for? There is no such parameter in the Jest documentation.

[uladzimirdev]

What is the "-i" parameter for?

https://jestjs.io/docs/cli#--runinband

[l0gicgate]

This addressed our issue as well with Jest constantly using more memory between tests. Unfortunately could not use workerIdleMemoryLimit setting as our runners are using Linux where memory isn't reported correctly.

I'm curious as to what the fix is, I've looked at the release and cannot find the commit that addresses this. It should be way more notable in the release notes that this was fixed

https://nodejs.org/en/blog/release/v21.1.0

Either way, very thankful this has been addressed after 2 years. Hoping for a backport to 20 and 18. We need this fix in an LTS version.

[joyeecheung]

I'm curious as to what the fix is, I've looked at the release and cannot find the commit that addresses this. It should be way more notable in the release notes that this was fixed

The commits that fixed it are in the vm: fix V8 compilation cache support for vm.Script part in that blog post, where Jest was also mentioned, though it may not be immediately obvious why. I think the issues described from the Jest side are multi-faceted and as #11956 (comment) mentioned this only mitigated the regressions for pure CJS compilation (i.e. regression comes back when there's actual import() or --experimental-vm-modules).

To summarize, the regression was introduced when Node.js implemented the importModuleDynamically option back in v16, to retrieve this option when import() is called within the script compiled, Node.js needs to set a host-defined option for this script. Previously, when V8 re-compiled a script that it has seen before, it could do a quick lookup on an internal table and return the script that's already compiled, making subsequent compilation almost free. The introduction of host-defined options resulted in cache misses, because in the implementation of Node.js up until not long ago, this option was different for every script compiled. The cache miss not only resulted in a performance regression (then V8 had to compile the previously-seen scripts from scratch every time), but could also contribute to higher memory usage (every script subsequently compiled is cached again with a slightly different option even though the source code remains the same). There was also another source of memory leak coming from how the scripts are managed within Node.js. To fix the issues, we worked around the problem in Node.js by:

1. Rework the memory management of the scripts and host-defined options in module: rework of memory management in vm APIs with the importModuleDynamically option nodejs/node#48510 which needed additional V8 API support that we upstreamed (see the PR for a detailed write-up of what was happening)
2. Use a constant host-defined option as the default when importModuleDynamically isn't configured vm: use default host-defined options when importModuleDynamically is not set nodejs/node#49950 - and because it's a constant for all scripts compiled without this option, the cache can be hit again in this case. However, this would still require some changes from Jest's side, so we did 3.
3. For Jest's use case, where importModuleDynamically is always configured but will throw a warning when --experimental-vm-modules isn't set (because then you can't actually do anything useful with that option anyway), we use another constant option to throw a built-in warning that serve the same purpose in vm: reject in importModuleDynamically without --experimental-vm-modules nodejs/node#50137, which finally leads to automatic fix of this issue in v21.1.0 without further changes from Jest's side.

So the issues are only mitigated, but not yet fully fixed - it comes back when --experimental-vm-modules is used. See nodejs/node#35375 (comment) about remaining issues & ideas about how they can be fixed/worked around, so it's still work in progress.

[SimenB]

Node 20 with backport coming Thursday: nodejs/node#50682

[hwildwood]

Node 20.10.0 has been released including the fixes mentioned in this PR 🎉

https://github.com/nodejs/node/releases/tag/v20.10.0

[AlecksJohannes]

We are close to victory folks 😆 , 1 approval at a time!

docker-library/official-images#15761

[karlhorky]

PR for Homebrew on macOS:

• node@20 20.10.0 Homebrew/homebrew-core#155156

[pwadmore-ea]

Please can I ask if the resolution will be backported to Node.js 18?

[Smrtnyk]

Please can I ask if the resolution will be backported to Node.js 18?

I don't think jest is the place where that gets decided

[karlhorky]

@pwadmore-ea this was asked over here in the Node.js repo:

• vm: reject in importModuleDynamically without --experimental-vm-modules nodejs/node#50137 (comment)

Seems like it's being backported here:

• [18.x] vm: backport vm-related fixes nodejs/node#51004

[kevinverelst]

Hi, I'm having the same problem but as far as I understand, version 20.11.0 of node should have this fixed?

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.