Handle inbound events from federation asynchronously #10272

erikjohnston · 2021-06-29T17:37:52Z

Based on #10269

This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous.

Two drawbacks:

Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately.
We now only process one event per room at a time. This can be fixed up further down the line.

erikjohnston · 2021-06-29T17:41:19Z

synapse/federation/federation_server.py

+
+        # The common path is for the event we just received be the only event in
+        # the room, so instead of pulling the event out of the DB and parsing
+        # the event we just pull out the next event ID and check if that matches.


FWIW I don't know whether the extra complexity is worth saving fetching a single event from the DB, happy to delete this logic 🤷

synapse/federation/federation_server.py

richvdh

looks generally about right

synapse/storage/schema/main/delta/59/16federation_inbound_staging.sql

synapse/federation/federation_server.py

synapse/storage/databases/main/event_federation.py

Co-authored-by: Patrick Cloke <[email protected]> Co-authored-by: Richard van der Hoff <[email protected]>

…ing.sql Co-authored-by: Richard van der Hoff <[email protected]>

Synapse 1.37.1rc1 (2021-06-29) ============================== Features -------- - Handle inbound events from federation asynchronously. ([\matrix-org#10269](matrix-org#10269), [\matrix-org#10272](matrix-org#10272))

This broke in #10272

Synapse 1.37.1 (2021-06-30) =========================== This release resolves issues (such as [matrix-org#9490](matrix-org#9490)) where one busy room could cause head-of-line blocking, starving Synapse from processing events in other rooms, and causing all federated traffic to fall behind. Synapse 1.37.1 processes inbound federation traffic asynchronously, ensuring that one busy room won't impact others. Please upgrade to Synapse 1.37.1 as soon as possible, in order to increase resilience to other traffic spikes. No significant changes since v1.37.1rc1. Synapse 1.37.1rc1 (2021-06-29) ============================== Features -------- - Handle inbound events from federation asynchronously. ([\matrix-org#10269](matrix-org#10269), [\matrix-org#10272](matrix-org#10272))

Synapse 1.37.1 (2021-06-30) =========================== This release resolves issues (such as [#9490](matrix-org/synapse#9490)) where one busy room could cause head-of-line blocking, starving Synapse from processing events in other rooms, and causing all federated traffic to fall behind. Synapse 1.37.1 processes inbound federation traffic asynchronously, ensuring that one busy room won't impact others. Please upgrade to Synapse 1.37.1 as soon as possible, in order to increase resilience to other traffic spikes. No significant changes since v1.37.1rc1. Synapse 1.37.1rc1 (2021-06-29) ============================== Features -------- - Handle inbound events from federation asynchronously. ([\#10269](matrix-org/synapse#10269), [\#10272](matrix-org/synapse#10272))

This was broken by matrix-org/synapse#10272

Synapse 1.37.1 (2021-06-30) =========================== This release resolves issues (such as [#9490](matrix-org/synapse#9490)) where one busy room could cause head-of-line blocking, starving Synapse from processing events in other rooms, and causing all federated traffic to fall behind. Synapse 1.37.1 processes inbound federation traffic asynchronously, ensuring that one busy room won't impact others. Please upgrade to Synapse 1.37.1 as soon as possible, in order to increase resilience to other traffic spikes. No significant changes since v1.37.1rc1. Synapse 1.37.1rc1 (2021-06-29) ============================== Features -------- - Handle inbound events from federation asynchronously. ([\#10269](matrix-org/synapse#10269), [\#10272](matrix-org/synapse#10272))

Synapse 1.37.1 (2021-06-30) =========================== This release resolves issues (such as [matrix-org#9490](matrix-org#9490)) where one busy room could cause head-of-line blocking, starving Synapse from processing events in other rooms, and causing all federated traffic to fall behind. Synapse 1.37.1 processes inbound federation traffic asynchronously, ensuring that one busy room won't impact others. Please upgrade to Synapse 1.37.1 as soon as possible, in order to increase resilience to other traffic spikes. No significant changes since v1.37.1rc1. Synapse 1.37.1rc1 (2021-06-29) ============================== Features -------- - Handle inbound events from federation asynchronously. ([\matrix-org#10269](matrix-org#10269), [\matrix-org#10272](matrix-org#10272))

erikjohnston changed the base branch from develop to erikj/async_federation_base_branch June 29, 2021 17:40

erikjohnston commented Jun 29, 2021

View reviewed changes

erikjohnston force-pushed the erikj/async_inbound_federation branch from f265174 to c2707da Compare June 29, 2021 17:42

clokep reviewed Jun 29, 2021

View reviewed changes

synapse/federation/federation_server.py Outdated Show resolved Hide resolved

richvdh reviewed Jun 29, 2021

View reviewed changes

erikjohnston and others added 6 commits June 29, 2021 19:32

Handle inbound federation transactions asynchronously

0cc2816

Newsfile

d541c74

Apply suggestions from code review

3d096ce

Co-authored-by: Patrick Cloke <[email protected]> Co-authored-by: Richard van der Hoff <[email protected]>

Remove room version in table

57e1f96

Update synapse/storage/schema/main/delta/59/16federation_inbound_stag…

5c94b10

…ing.sql Co-authored-by: Richard van der Hoff <[email protected]>

Update blacklist with now failing tests

f43cfe8

richvdh force-pushed the erikj/async_inbound_federation branch from 280baea to f43cfe8 Compare June 29, 2021 18:32

richvdh merged commit c54db67 into erikj/async_federation_base_branch Jun 29, 2021

richvdh deleted the erikj/async_inbound_federation branch June 29, 2021 18:55

richvdh mentioned this pull request Jun 29, 2021

followup on asynchronous-federation work #10275

Closed

9 tasks

erikjohnston mentioned this pull request Jun 30, 2021

Fix 'Backfilled events whose prev_events are in a different...' test matrix-org/sytest#1059

Merged

erikjohnston mentioned this pull request Jun 30, 2021

Fix the inbound PDU metric #10279

Merged

erikjohnston added a commit that referenced this pull request Jun 30, 2021

Fix the inbound PDU metric

7f96eff

This broke in #10272

erikjohnston added a commit that referenced this pull request Jun 30, 2021

Fix the inbound PDU metric (#10279)

329ef5c

This broke in #10272

richvdh mentioned this pull request Jul 2, 2021

Synapse appears to be sending the next transaction while a transaction has yet to be processed #10260

Closed

erikjohnston added a commit to matrix-org/complement that referenced this pull request Jul 5, 2021

Fix race in MSC3083 TestRestrictedRoomsRemoteJoin

4e021cd

This was broken by matrix-org/synapse#10272

erikjohnston mentioned this pull request Jul 5, 2021

Fix race in MSC3083 TestRestrictedRoomsRemoteJoin matrix-org/complement#149

Merged

erikjohnston added a commit to matrix-org/complement that referenced this pull request Jul 7, 2021

Fix race in MSC3083 TestRestrictedRoomsRemoteJoin (#149)

76db9e5

This was broken by matrix-org/synapse#10272

erikjohnston mentioned this pull request Jul 26, 2021

Federation from matrix.org to half-shot.uk taken out for several hours due tombstone event (and subsequent membership spam) in room. #10047

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle inbound events from federation asynchronously #10272

Handle inbound events from federation asynchronously #10272

erikjohnston commented Jun 29, 2021

erikjohnston Jun 29, 2021

richvdh Jun 29, 2021

richvdh left a comment

Handle inbound events from federation asynchronously #10272

Handle inbound events from federation asynchronously #10272

Conversation

erikjohnston commented Jun 29, 2021

erikjohnston Jun 29, 2021

Choose a reason for hiding this comment

richvdh Jun 29, 2021

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment