This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Yield during large v2 state res. #7735
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
State res v2 across large data sets can be very CPU intensive, and if all the relevant events are in the cache the algorithm will run from start to finish within a single reactor tick. This can result in blocking the reactor tick for several seconds, which can have major repercussions on other requests. To fix this we simply add the occaisonal `sleep(0)` during iterations to yield execution until the next reactor tick. The aim is to only do this for large data sets so that we don't impact otherwise quick resolutions.
auscompgeek
reviewed
Jun 23, 2020
clokep
reviewed
Jun 23, 2020
auscompgeek
reviewed
Jun 24, 2020
synapse/state/v2.py
Outdated
| yield _add_event_and_auth_chain_to_graph( | ||
| graph, room_id, event_id, event_map, state_res_store, auth_diff | ||
| ) | ||
|
|
||
| # We yield occasionally when we're working with large data sets to | ||
| # ensure that we don't block the reactor loop for too long. | ||
| if idx != 0 and idx % _YIELD_AFTER_ITERATIONS == 0: |
There was a problem hiding this comment.
Idea: instead of checking idx != 0, why not make enumerate start at 1? The first yield would be 1 iteration early, but that's probably an acceptable tradeoff for avoiding a comparison on every iteration.
There was a problem hiding this comment.
Aha, didn't realise that was a thing enumerate supports
babolivier
pushed a commit
that referenced
this pull request
Sep 1, 2021
* commit 'dc80a0762': 1.16.0rc1 Back out MSC2625 implementation (#7761) Additional configuration options for auto-join rooms (#7763) Add some metrics for inbound and outbound federation processing times (#7755) Explain the purpose of the "tests" conditional dependency requirement (#7751) Add another yield point to state res v2 (#7746) Move flake8 to end. Don't exit script on failure (#7738) Make tox actions work on Debian 10 (#7703) Yield during large v2 state res. (#7735) add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675) Set Content-Length for Metrics requests (#7730) Sync ignored table names in synapse_port_db to current database schema (#7717) Allow local media to be marked as safe from being quarantined. (#7718) Convert directory handler to async/await (#7727) Speed up state res v2 across large state differences. (#7725)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.
To fix this we simply add the occaisonal
sleep(0)during iterations toyield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.