waitFor + getByRole causing severe delays

[herecydev]

• @testing-library/dom version: 7.26.6
• node version: 14.3.0

Relevant code or config:

it("Causes a big delay", async () => {
      // This fires a fetch request that is mocked using MSW (so it should be near instant)
      userEvent.click(screen.getByRole("button", { name: "Search"}));

      await waitFor(() => {
        //super fast call
        //screen.getByText("some_link_text");

        // causes massive delays
        screen.getByRole("link", { name: "some_link" });
      });
    });

What you did:

Running this test causes the fetch request promise to resolve a lot longer than usual (around 1.5 - 2s). I initially investigated this by wrapping console.time around the request.

I also had an additional test which looks very similar to the above, but instead of using getByRole it was using getByText to search for some text. That test took a fraction of the time:

What happened:

After changing getByRole to getByText or inserting a getByText query before the getByRole it resolved much faster. My working theory is that getByRole is so expensive that it was monopolising the waitFor loop and takes a long time to relinquish control to the pending promise that needs to resolve.

[nickserv]

The issue is that getByRole is fairly slow, but it's also syncronous, so waitFor keeps rerunning it and has to wait for each call to finish before other code can run. It could potentially use some performance improvements, but perhaps we could make it asyncronous (or add an alternative for back compatibility) so this can be run asyncronously.

[herecydev]

Thanks for the quick reply Nick, does that not suggest that any waitFor that is sufficiently slow would trigger this? If so, is waitFor problematic in it's design?

What do you think the best short term resolution for this?

[nickserv]

I'm guessing waitFor may be so eager to quickly rerun longer functions that it might actually be slowed down. You could try increasing the interval option, it defaults to only 50 milliseconds.

[herecydev]

For reference the getByRole was taking 400-500ms and would cause the waitFor to loop 3-4 times. The best fix (for time) is to insert a more trivial query before that (like a getByText) as that flushes that promise much faster.

Would you like me to document some of this? Feels like a sufficiently frustrating morning that I don't want others to go through!

[nickserv]

A higher interval might cause less loops. Even if it doesn't make this specific test run shorter, a longer interval could allow other tests to run in parallel more effectively. Can you try this out?

Alternatively, we could dynamically adjust the interval by default based on how long it takes each time. I'm not sure if that's worth the effort though.

[herecydev]

Setting an interval of 750 still caused 2 iterations to fail (~450ms), then the fetch promise resolves. There's one more failure (output still not rendered to the screen), finally it passes.

Setting an interval underneath the getByRole time causes it to fail as expected. I.e. setting it to 200ms will cause 3 iterations and then the waitFor will just give up (total time is 1s by default if I remember).

I guess it works, but you keep having to up this arbitrary interval to "match" however long the waitFor will take as the complexity grows. That doesn't feel great to me.

[nickserv]

Yea, I was thinking maybe waitFor could intelligently learn how long previous calls took and use that to adjust the interval. I'm not sure how well it would work in practice though. On the other hand, you may not need *byRole (see #698 (comment)), though PRs to improve performance would be welcome.

[herecydev]

The problem is when the knowledge has been propogated differently;

https://kentcdodds.com/blog/common-mistakes-with-react-testing-library#using-the-wrong-query
https://testing-library.com/docs/guide-which-query/#priority

There's absolutely nothing in the docs about the downsides of ByRole, and previously I've been happy to accept the small performance hit for the better testing methodology. But this issue is straight out problematic now.

I would love to get an action on how best proceed with this, do we:

1. auto-increment waitFor intelligently
2. Change inner workings of ByRole to be asynchronous
3. Provide documentation around this known issue with suitable guidance

etc.

[nickserv]

1 would be cool to experiment with but as I said previously, I'm not sure how difficult to implement and useful it would be, so it may not be worth it. I'm fine with either 2 or 3.

@kentcdodds @eps1lon What do you think about an asyncronous version of *byRole (2) to work around performance issues like this?

[kentcdodds]

One question to make sure I understand. What is your test waiting on that takes more than a single 50ms interval? Most/All tests using this utility should mock out any async code so it resolves on the next tick or two of the event loop 🤔

[kentcdodds]

Just noticed the screenshot, sorry about that. Looks like it's getting mocked by MSW, so I wouldn't expect this to take longer than a few milliseconds. Something is up there.

The *ByRole query is definitely the performance hog of the bunch, but it's a cost with the benefit of confidence which is why it's the top most recommended query. I suppose we could add a note to the docs warning about this. Probably a good idea. I'd love to make that query faster, but I still recommend it.

In this situation, there's something especially fishy going on. I think we'd need a reproduction to dive any deeper though.

[herecydev]

@kentcdodds yer to confirm MSW is registering is ~1ms so it's very fast. It's the *ByRole that's exceeding the waitFor interval window.

Reproduction will take some time, is it not clear that exceeding the waitFor interval window will cause this?

[eps1lon]

If you want to optimize tests for performance go through these steps until you're satisfied. They're ordered by their confidence impact (lower impact, higher ranked) then perf impact (higher impact, higher ranked).

1. Make sure you use the latest version of jsdom
2. don't use JSDOM but modern browsers
3. getByRole(role, { hidden: true }) (WARNING: includes inaccessible elements - avoids expensive visibility check)
4. getByTestId instead of getByRole
5. don't filter by name in getByRole (Can rely on order in getAllByRole. Order of elements in the DOM is in most cases not an implementation detail.)

Not using jsdom at all is the best solution since it has better perf and higher confidence.

[kentcdodds]

@eps1lon great walkthrough 👍 though I think I'd prefer to try another query before testIDs. I think it makes sense to add this to the docs.

[eps1lon]

though I think I'd prefer to try another query before testIDs. I think it makes sense to add this to the docs.

Definitely! It was easier to say since "use another query" might not be that helpful. If you're really blocked by perf problems I would just suggest a quick escape hatch and revisit it once the test breaks on unrelated changes (though that might be even harder to judge). It's just really hard to give actionable advise that people can apply to different scenarios.

I think the most generic advise is to re-evaluate your waitFor usage. If you're in a unit-test-like environment you should be able to mock the timers so that you can deterministically forward to the point where your assertion should be true (skipping the wasteful polling approach). In e2e tests (with real timers/requests) I'd like to see a reproduction since I would expect a browser to not have perf problems with getByRole(role, { name }).

And to give an optimistic outlook: The ByRole implementation is just a polyfill at the moment. Long term we'll use actual DOM APIs that are hopefully faster (see https://wicg.github.io/aom/spec/#computed-accessibility-tree).

[alexkrolick]

I would definitely try item 3 from @eps1lon's checklist for this particular case - the visibility calculation is expensive in large DOM trees

getByRole(role, { hidden: true })

[kentcdodds]

Thanks for the input here everyone. I think some docs here would go a long way.

@herecydev once you've tried and had success with one of these recommendations, would you mind working on improvement the documentation to help others who may face this in the future?

[eps1lon]

I would definitely try item 3 from @eps1lon's checklist for this particular case - the visibility calculation is expensive in large DOM trees

To touch on another potential optimization point: In Material-UI we set the default value to true (fast option) in local development and to false (slow) in CI. The reason being that you rarely accidentally query for inaccessible elements. If you do, it should be obvious in the test which we'll check during code review (since an explicit hidden option stands out). In 1015 ByRole calls we only ever need to explicitly specify the hidden state in 25 cases. So in the vast majority of cases we just check the hidden state for confidence not because the test is actually about the hidden state.

The other part is that CI can be a little bit slower (since it already takes a couple of minutes). However, for local development you want snappy feedback so every millisecond counts.

That might be worth documenting. Though we shouldn't integrate that into the library since we don't know whether people have CI or if every commit passes through it.

[herecydev]

If you want to optimize tests for performance go through these steps until you're satisfied. They're ordered by their confidence impact (lower impact, higher ranked) then perf impact (higher impact, higher ranked).

1. Make sure you use the latest version of `jsdom`

2. don't use JSDOM but modern browsers

3. `getByRole(role, { hidden: true })` (WARNING: includes inaccessible elements - avoids expensive visibility check)

4. `getByTestId` instead of `getByRole`

5. don't filter by `name` in `getByRole` (Can rely on order in `getAllByRole`. Order of elements in the DOM is in most cases not an implementation detail.)

Not using jsdom at all is the best solution since it has better perf and higher confidence.

Thanks Sebastian, to address these;

1. On the latest version
2. Can you elaborate on this one, we're using RTL with jest (and JSDOM). Is there a way to configure other environments, if so what would you recommend?
3. We're already setting this to true globally
4. I've already mentioned that I'm getting mixed messages with this one. @kentcdodds has clearly called this out as a common mistake: https://kentcdodds.com/blog/common-mistakes-with-react-testing-library#using-the-wrong-query
5. Similar to above, I want to look for a link that also has some text. Is this not preferable to combine into a single query

I think the most generic advise is to re-evaluate your waitFor usage. If you're in a unit-test-like environment you should be able to mock the timers so that you can deterministically forward to the point where your assertion should be true (skipping the wasteful polling approach). In e2e tests (with real timers/requests) I'd like to see a reproduction since I would expect a browser to not have perf problems with getByRole(role, { name }).

I just want to stress this is nothing to do with timers. Everything is mocked. We're using MSW here, but the *ByRole query is taking longer than the waitFor and so waitFor times out.

[alexkrolick]

Can you elaborate on this one, we're using RTL with jest (and JSDOM). Is there a way to configure other environments, if so what would you recommend?

ie Puppeteer, Cypress

How big is the DOM tree being queried in this test?

[meastes]

I was seeing similar performance issues in one of my tests as well. My app is using Material UI and I was seeing about 6 seconds per query with getByRole() making my total test time about 20 seconds. After switching the queries to getByLabelText() the test time went down to 600ms - HUGE difference. The component under test is a Drawer with about 50 checkboxes in it.

I agree that a small performance cost is worth the confidence, but it becomes unusable when that cost is about 6 seconds. I definitely agree in having some documentation updates to give visibility into this tradeoff.

[eps1lon]

I just want to stress this is nothing to do with timers. Everything is mocked.

Why are you using waitFor if everything is mocked? If everything is mocked a single jest.runAllTimers should be faster than waitFor. Could you run npx testing-library-envinfo react and share the results?

Can you elaborate on this one, we're using RTL with jest (and JSDOM). Is there a way to configure other environments, if so what would you recommend?

We're using mocha + karma with chrome headless and browserstack. Bundling the full test suite with webpack and running is as fast as running it with JSDOM. So there's a breakpoint where mocha + karma becomes faster if you get bundle time low enough. We don't have as severe problems with JSDOM so I suspect that this might be a real time saver for you.

However, I don't think it's appropriate for local development where you just run a subset of the tests.

I was seeing about 6 seconds per query with getByRole()

6 full seconds for each call of getByRole? Could you run npx testing-library-envinfo react and share the results? If you could share a repro I might be able to understand the issue. Having a repro for these problems is important since that's the only way we can meaningfully work on the performance. Make sure you

[shan-du]

We are having similar performance issues with getByRole.
I did some comparison fetching the exact same element using getByText usually takes around 200~300ms, but with getByRole it takes upwards of 40 seconds!!!

I know that both of these durations are a bit long anyways due to the DOM we are testing, but the point here is that relative to each other, getByRole is significantly slower, which is just ridiculous.

In the end I had to give the selectors a container scope like container.getByRole instead of screen.getByRole, then the amount of time reduces drastically, but in comparison, it is still a lot slower than getByText.

I'm really not sure what the reason is and perhaps it should really be mentioned in the documentations.

[herecydev]

@eps1lon sorry I can't be of much help now, this issue was quite painful and we move over to using cypress where this isn't a problem for us, as well as providing a suite of other benefits.

[eps1lon]

I did some comparison fetching the exact same element using getByText usually takes around 200~300ms, but with getByRole it takes upwards of 40 seconds!!!

@shan-du Please share a reproducible example. Otherwise we can't help you with this problem.

[alexkrolick]

I wonder if it has something to do with the MutationObserver optimizations in waitFor hogging too much of the thread for the more expensive getByRole DOM traversal

[meastes]

Here's a reproducible example. It's not quite as bad as my original project (about 5 seconds per getByRole call in the example instead of 6 seconds), but still shows that there's a huge performance issue. Hopefully this is helpful in debugging the issue!

https://github.com/meastes/get-by-role-example

 PASS  src/components/DrawerExample.test.tsx (22.395 s)
  DrawerExample
    selecting
      ✓ defaults checkboxes to checked when selectedColumns is specified (840 ms)
      ✓ defaults checkboxes to checked when selectedColumns is specified (getByRole) (17729 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        23.846 s, estimated 30 s
Ran all test suites matching /DrawerExample/i.

[cmacdonnacha]

Looking for some opinions. Is the byRole query as vital when using a UI library like MUI, who pride themselves on having fully accessible components? We're really starting to see significant performance issues within our tests and I have always promoted the use of byRole to ensure our code is accessibility friendly. However, if we can "trust" that these checks are being done for the most part by MUI, we may look into using byText more regularly in tests where performance is hit the most.

[mmnoo]

I've already started replacing byRole. It costs too much. byText isnt the same, yes, but it will have to do. Balancing tradeoffs.

[MatanBobi]

@cmacdonnacha, @mmnoo I gave a talk to explain what's happening in ByRole query to understand what's happening as part of it and understand why it may still have a purpose. I recommend giving it a look (though I'm a bit biased):
https://www.youtube.com/watch?v=jNAMdsbdvlIs&ab_channel=ReactNext

[frankandrobot]

@MatanBobi Love what you're doing with this library. But want to apologize in advance for possibly coming off as to harsh.

Here's the video summary:

getByRole is super complex under the hood---that's why it's so slow. Basically, you're saying that "it is what it is".

• question: is it feasible to create a "light" version of getByRole that makes stronger assumptions on the DOM tree and is possibly faster? (ex: Not an a11y specialist so don't know if this makes sense but I would be OK w/ explicitly tagging elements with roles)
• FYI---getting CI under control is of utmost importance so after seeing this video, I'm now in the camp of not recommending getByRole. As others stated, "tradeoffs"
• workarounds? a real browser? What about changing the run time---bun or Hermoine?

[MatanBobi]

@frankandrobot, you're not, and we're 100% open for criticism.
I'm not saying it is what it is, we are trying to workaround those, having said that, the major bottleneck here is usually JSDOM.
Regarding your specific question:

1. If you don't care about visibility check for example, you can use hidden for example to avoid that check and get a tiny speedup. The rest is probably too related to the role selection and we can't remove it.
2. I think that not using ByRole misses a bit of the accessibility checks and misses the point of our guiding principle.
3. I highly recommend trying vitest instead of jest and also happy-dom instead of jsdom for some quick wins :)
   Feel free to ask more questions, I'll do my best to help.

[frankandrobot]

We've used vitetest before While it does help test boot up, if the problem is Jsdom, it doesn't provide speed ups to slow queries. As for happy Dom, it's too incomplete to use in production. I really wish you guys would stop recommending it 😜 On Nov 6, 2023 11:01 AM, Matan Borenkraout ***@***.***> wrote: @frankandrobot<https://github.com/frankandrobot>, you're not, and we're 100% open for criticism. I'm not saying it is what it is, we are trying to workaround those, having said that, the major bottleneck here is usually JSDOM. Regarding your specific question: 1. If you don't care about visibility check for example, you can use hidden<https://testing-library.com/docs/queries/byrole#hidden> for example to avoid that check and get a tiny speedup. The rest is probably too related to the role selection and we can't remove it. 2. I think that not using ByRole misses a bit of the accessibility checks and misses the point of our guiding principle. 3. I highly recommend trying vitest instead of jest and also happy-dom instead of jsdom for some quick wins :) Feel free to ask more questions, I'll do my best to help. — Reply to this email directly, view it on GitHub<#820 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAGAGMNJ4OATI5N6BVALD2DYDEJXFAVCNFSM4TUQYKQKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZZGU2DSOJZGU2A>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[MatanBobi]

I have used vitest and happy-dom for production use as a replacement to jest + jsdom or else I wouldn't have recommended it. I can understand why some cases are still not supported, but you can definitely migrate some tests and run them side by side.

[frankandrobot]

Yes. I'm not saying it's not usable. But it's by no means a complete replacement. To give an example, in my code base, in about a dozen test files. We're only able to migrate two to happy Dom. That's too low to be a useful replacement. On Nov 6, 2023 11:10 AM, Matan Borenkraout ***@***.***> wrote: I have used vitest and happy-dom for production use as a replacement to jest + jsdom or else I wouldn't have recommended it. I can understand why some cases are still not supported, but you can definitely migrate some tests and run them side by side. — Reply to this email directly, view it on GitHub<#820 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAGAGMPRM7DF4BVNBJGNZC3YDEKXTAVCNFSM4TUQYKQKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZZGU2TCOJYG4YA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[sibelius]

what is the workaround ?

[eps1lon]

what is the workaround ?

#820 (comment)

[sibelius]

how to migrate from jsdom to a real browser ?

we are using jest + testing library

is there an environment for this ?

[eps1lon]

how to migrate from jsdom to a real browser ?

This is out of scope for this discussion. Migrating from Jest+JSDOM to Playwright is not trivial. I'd suggest looking up Playwright's Getting Started docs to get a feeling how testing works in Playwright vs. Jest. In Playwright you wouldn't need @testing-library/react since the queries are built-into Playwright.

[sibelius]

is there a faster jsdom ?

[melan0802]

@eps1lon Have you considered replace filter with find in queryAllByRole when we use getByRole?

dom-testing-library/src/queries/role.ts

Lines 266 to 281 in 56543d5

	.filter(element => {
	if (name === undefined) {
	// Don't care
	return true
	}
	return matches(
	computeAccessibleName(element, {
	computedStyleSupportsPseudoElements:
	getConfig().computedStyleSupportsPseudoElements,
	}),
	element,
	name as MatcherFunction,
	text => text,
	)
	})

It will allow us to skip some expensive computeAccessibleName after we find our tartget.

[Verthon]

is there a faster jsdom ?

You can try happy-dom https://github.com/capricorn86/happy-dom it doesn't have some Web APIs but maybe it will be enough for you.
Quick benchmark:
#820 (comment)

[foxaltus]

Hey! I want to share my experience here as I've been struggling with a few slow tests that were consistently timing out.

In my case, the slow tests were the result of using screen.findByRole("cell", { name: 'xxx' }) on tables (with less than 100 cells).

I tried migrating to happy-dom, which was pretty much effortless (minus a few issues caused by poorly written tests on my side). However, I only obtained marginal gains with happy-dom. Tests that were timing out with jest-dom were also timing out with happy-dom.

I tried using the hidden option too (or the global defaultHidden) of testing-library with marginal gains again, but not enough to solve the timeout.

The only thing that worked for me was to replace findByRole with findByText. Not ideal, but it makes a huge difference.

I am not sure if there is a particular issue with tables, but I would advise to stay away from accessible queries when working with them...

[TomPridham]

@eps1lon Have you considered replace filter with find in queryAllByRole when we use getByRole?

dom-testing-library/src/queries/role.ts

Lines 266 to 281 in 56543d5

	.filter(element => {
	if (name === undefined) {
	// Don't care
	return true
	}
	return matches(
	computeAccessibleName(element, {
	computedStyleSupportsPseudoElements:
	getConfig().computedStyleSupportsPseudoElements,
	}),
	element,
	name as MatcherFunction,
	text => text,
	)
	})

It will allow us to skip some expensive computeAccessibleName after we find our tartget.

i don't think we want to do that because then the single element matchers wouldn't throw if there was more than one element

[frankandrobot]

We've been exploring a possible workaround. Here's some context:

• An element's inferred name is the inferred aria label. Ex: <button>A button</button> has an inferred aria label of "a button". There are also inferred roles. The button above has a role of "button", but almost anything can have a button role by just assigning role attribute a value of "button".
• getByRole has quicker cousins, namely getByText and getByAriaLabel. (However, by themselves they don't check for role.)
• The workaround is to use one of these cousins to search by label or inferred name and then use within on the parent element to assert the role. In testing, this has been almost as fast as plain getByText!

But of course there are caveats:

1. Users need to know whether or not the aria-label is inferred.
2. The inferred name can only be in the top-level of the component. Otherwise, selection will fail. For example, in <td><div>a cell</div></td>, the inferred name of the cell is "a cell", so the workaround fails---it gets the DIV and then checks whether or not it's a cell, so it fails.
3. The role check is done by calling within#getAllByRole (w/ no name) on the parent element. In edge cases, it's DOM tree can be just as complex as the root DOM tree, so there will be no speed up. (It may be possible to improve on this by temporarily detaching the node from the DOM.)

Notes:

1. Yes, you do need to know implementation details but IMO it isn't a big of a deal. It's a bigger issue w/ third party libraries but IMO there should be test selector helpers for these anyway.
2. At least in our case, we use a shared component library, so instances of caveat 1) are limited to a handful of cases.

The code:

type GetByRoleFullOpts = Parameters<typeof original.getByRole>;
type GetByRoleOpts = GetByRoleFullOpts[1] & { nameType?: "inferred" | "aria-label" }

export const getByRole = (role: GetByRoleFullOpts[0], opts: GetByRoleOpts): GetByRoleReturn => {
  const { name, nameType, ...restOpts } = opts ?? {};

  if (name && typeof name !== "function") {
    const getter = nameType === "inferred" ? original.getByText : original.getByLabelText;
    const foundElement = getter(name, restOpts);
    const elements = within(foundElement.parentElement ?? document.documentElement).getAllByRole(role, restOpts);
    const match = elements.find((elem) => elem === foundElement);
    expect(match).toBeDefined();
    return foundElement;
  } else {
  return original.getByRole(role, opts);
};

[benlesh]

I've profiled issues with this in our codebase locally, and 90% of the speed issues are coming from checking to see if the element's subtree is "accessible", meaning if it's potentially visible to the user. The problem is it recursively checks every element, and worse, runs a getComputedStyle on it to see if its visible.

There's a new API for that now checkVisibility, but that's probably not going to help much in node.

It's probably prudent to add an escape hatch here to skip this check. something like getAllByRole('option', { strictVisibilityCheck: false })

Honestly, I think even having this check to begin with is a bit pedantic given the nature of these tests. If it was my project, I would default to not even checking that unless someone had it on getAllByRole('option', { strictVisibilityCheck: true }) or the like.

[MatanBobi]

Thanks @benlesh for the investigation.
This is already an option as stated by @eps1lon in this comment by using getByRole('option', { hidden: true }), which will skip the isInaccessible checks. This can also be turned off by default by calling configure({defaultHidden: true}), more info in our docs.

[hwride]

Last I checked hidden: true was not a particularly great improvement for performance of getByRole. So even if it does skip the accessibility checks, it doesn't seem to be the whole story/solution.

For us getByText or getting by a specific role attribute was a better trade off. Yeah you might have to tweak a selector without implicit role finding from time to time, but in reality those changes are very infrequent and small compared to the rest of time spent on test writing, and well worth the performance benefit.

[benlesh]

For record keeping, I did submit a PR: #1331

I'll explore other options for our needs. These tests are honestly slower than just running Playwright or Cypress tests at this point.

[benlesh]

@hwride you might want to check again. hidden: true improved performance of getAllByRole at least 10x in my problem test.

[eps1lon]

I'll explore other options for our needs. These tests are honestly slower than just running Playwright or Cypress tests at this point.

That is what we recommend for high confidence testing. For lower confidence testing, disabling the inaccessible check via hidden: true is recommended for local testing and then enabling this check in CI.

There's a new API for that now checkVisibility, but that's probably not going to help much in node.

Keep in mind that the slowness is mostly not coming from the implementation but the environment. Switching to checkVisibility will likely have no effect in Node.js since implementations of the DOM in Node.js will need to run style calculation anyway. This may improve performance in Browsers though. I left an issue in eps1lon/dom-accessibility-api#1060 for anybody to explore.

Node.js implementations of the DOM (e.g. JSDOM) are just slower than browsers and less feature complete. Both of these statements apply especially to CSS in Node.js which is why the library is perceived to be slow.