Improve perf of sync device lists #17216
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It's almost always more efficient to query the rooms that have device list changes, rather than looking at the list of all users whose devices have changed and then look for shared rooms.
erikjohnston
force-pushed
the
erikj/device_list_sync_perf
branch
from
2702f59 to
6d691c1
Compare
erikjohnston
force-pushed
the
erikj/device_list_sync_perf
branch
from
6d691c1 to
e6d3d80
Compare
erikjohnston
changed the title
reivilibre
reviewed
May 18, 2024
synapse/replication/tcp/client.py
Outdated
| @@ -112,6 +112,14 @@ async def on_rdata( | |||
| token: stream token for this batch of rows | |||
| rows: a list of Stream.ROW_TYPE objects as returned by Stream.parse_row. | |||
| """ | |||
| all_room_ids: Set[str] = set() | |||
| if stream_name == DeviceListsStream.NAME: | |||
| prev_token = self.store.get_device_stream_token() | |||
There was a problem hiding this comment.
not that familiar as always, but it sounds unintuitive to get the latest stream token and call it prev_token — what's going on here?
There was a problem hiding this comment.
Ah, this really needs a comment. Basically at this point we haven't updated the id generators, and so they don't know about the new stuff that has just come in (that happens in process_replication_position). What we're basically doing is getting an ID of "this is where we're currently at", so that we can get the deltas from the DB for what has changed since then upto the new token.
There was a problem hiding this comment.
yup OK I was hoping it was something like that, but a comment would be good to avoid wtfery
reivilibre
approved these changes
May 18, 2024
| min_stream_id = await self._get_min_device_lists_changes_in_room() | ||
|
|
||
| if min_stream_id > from_id: | ||
| raise Exception("stream ID is too old") |
There was a problem hiding this comment.
not really materially important, but feels like this should be a more specific exception type
This was referenced May 22, 2024
Mic92
pushed a commit
to Mic92/synapse
that referenced
this pull request
Jun 14, 2024
Re-introduces element-hq#17191, and includes element-hq#17197 and element-hq#17214 The basic idea is to stop calling `get_rooms_for_user` everywhere, and instead use the table `device_lists_changes_in_room`. Commits reviewable one-by-one.
yingziwu
added a commit
to yingziwu/synapse
that referenced
this pull request
Jun 26, 2024
- Fix the building of binary wheels for macOS by switching to macOS 12 CI runners. ([\#17319](element-hq/synapse#17319)) - When rolling back to a previous Synapse version and then forwards again to this release, don't require server operators to manually run SQL. ([\#17305](element-hq/synapse#17305), [\#17309](element-hq/synapse#17309)) - Use the release branch for sytest in release-branch PRs. ([\#17306](element-hq/synapse#17306)) - Fix bug where one-time-keys were not always included in `/sync` response when using workers. Introduced in v1.109.0rc1. ([\#17275](element-hq/synapse#17275)) - Fix bug where `/sync` could get stuck due to edge case in device lists handling. Introduced in v1.109.0rc1. ([\#17292](element-hq/synapse#17292)) - Add the ability to auto-accept invites on the behalf of users. See the [`auto_accept_invites`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#auto-accept-invites) config option for details. ([\#17147](element-hq/synapse#17147)) - Add experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync/e2ee` endpoint for to-device messages and device encryption info. ([\#17167](element-hq/synapse#17167)) - Support [MSC3916](matrix-org/matrix-spec-proposals#3916) by adding unstable media endpoints to `/_matrix/client`. ([\#17213](element-hq/synapse#17213)) - Add logging to tasks managed by the task scheduler, showing CPU and database usage. ([\#17219](element-hq/synapse#17219)) - Fix deduplicating of membership events to not create unused state groups. ([\#17164](element-hq/synapse#17164)) - Fix bug where duplicate events could be sent down sync when using workers that are overloaded. ([\#17215](element-hq/synapse#17215)) - Ignore attempts to send to-device messages to bad users, to avoid log spam when we try to connect to the bad server. ([\#17240](element-hq/synapse#17240)) - Fix handling of duplicate concurrent uploading of device one-time-keys. ([\#17241](element-hq/synapse#17241)) - Fix reporting of default tags to Sentry, such as worker name. Broke in v1.108.0. ([\#17251](element-hq/synapse#17251)) - Fix bug where typing updates would not be sent when using workers after a restart. ([\#17252](element-hq/synapse#17252)) - Update the LemonLDAP documentation to say that claims should be explicitly included in the returned `id_token`, as Synapse won't request them. ([\#17204](element-hq/synapse#17204)) - Improve DB usage when fetching related events. ([\#17083](element-hq/synapse#17083)) - Log exceptions when failing to auto-join new user according to the `auto_join_rooms` option. ([\#17176](element-hq/synapse#17176)) - Reduce work of calculating outbound device lists updates. ([\#17211](element-hq/synapse#17211)) - Improve performance of calculating device lists changes in `/sync`. ([\#17216](element-hq/synapse#17216)) - Move towards using `MultiWriterIdGenerator` everywhere. ([\#17226](element-hq/synapse#17226)) - Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`. ([\#17229](element-hq/synapse#17229)) - Change the `allow_unsafe_locale` config option to also apply when setting up new databases. ([\#17238](element-hq/synapse#17238)) - Fix errors in logs about closing incorrect logging contexts when media gets rejected by a module. ([\#17239](element-hq/synapse#17239), [\#17246](element-hq/synapse#17246)) - Clean out invalid destinations from `device_federation_outbox` table. ([\#17242](element-hq/synapse#17242)) - Stop logging errors when receiving invalid User IDs in key querys requests. ([\#17250](element-hq/synapse#17250)) * Bump anyhow from 1.0.83 to 1.0.86. ([\#17220](element-hq/synapse#17220)) * Bump bcrypt from 4.1.2 to 4.1.3. ([\#17224](element-hq/synapse#17224)) * Bump lxml from 5.2.1 to 5.2.2. ([\#17261](element-hq/synapse#17261)) * Bump mypy-zope from 1.0.3 to 1.0.4. ([\#17262](element-hq/synapse#17262)) * Bump phonenumbers from 8.13.35 to 8.13.37. ([\#17235](element-hq/synapse#17235)) * Bump prometheus-client from 0.19.0 to 0.20.0. ([\#17233](element-hq/synapse#17233)) * Bump pyasn1 from 0.5.1 to 0.6.0. ([\#17223](element-hq/synapse#17223)) * Bump pyicu from 2.13 to 2.13.1. ([\#17236](element-hq/synapse#17236)) * Bump pyopenssl from 24.0.0 to 24.1.0. ([\#17234](element-hq/synapse#17234)) * Bump serde from 1.0.201 to 1.0.202. ([\#17221](element-hq/synapse#17221)) * Bump serde from 1.0.202 to 1.0.203. ([\#17232](element-hq/synapse#17232)) * Bump twine from 5.0.0 to 5.1.0. ([\#17225](element-hq/synapse#17225)) * Bump types-psycopg2 from 2.9.21.20240311 to 2.9.21.20240417. ([\#17222](element-hq/synapse#17222)) * Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425. ([\#17260](element-hq/synapse#17260))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-introduces #17191, and includes #17197 and #17205
The basic idea is to stop calling
get_rooms_for_usereverywhere, and instead use the tabledevice_lists_changes_in_room.Commits reviewable one-by-one.