Add support for MSC2716 marker events (federation)

[MadLittleMods]

Add support for "marker" events which makes historical events discoverable for servers that already have all of the scrollback history.

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

Follow-up to #9247 and #10245

Mermaid live editor playground link

mermaid graph syntax

flowchart BT
    subgraph live
        marker1>"marker"] ----> B -----------------> A
    end
    
    subgraph chunk0
        chunk0-chunk[["chunk"]] --> chunk0-2(("2")) --> chunk0-1((1)) --> chunk0-0((0)) --> chunk0-insertion[/insertion\]
    end

    subgraph chunk1
        chunk1-chunk[["chunk"]] --> chunk1-2(("2")) --> chunk1-1((1)) --> chunk1-0((0)) --> chunk1-insertion[/insertion\]
    end
    
    subgraph chunk2
        chunk2-chunk[["chunk"]] --> chunk2-2(("2")) --> chunk2-1((1)) --> chunk2-0((0)) --> chunk2-insertion[/insertion\]
    end

    
    marker1 -.-> chunk0-insertionBase
    chunk0-insertionBase[/insertion\] -------------> A
    chunk0-chunk -.-> chunk0-insertionBase[/insertion\]
    chunk1-chunk -.-> chunk0-insertion
    chunk2-chunk -.-> chunk1-insertion

Loading

---

TARDIS visualization: matrix-org/tardis#1

Dev notes

• event_backward_extremities

hs2 asks for the list of /messages

Call stacks for backfilling

From the remote server

• RoomMessageListRestServlet.on_GET: /rooms/(?P<room_id>[^/]*)/messages$ (hs2 asks for more messages)
• get_messages
• maybe_backfill
• _maybe_backfill_inner
  • get_oldest_events_with_depth_in_room
  • try_backfill
• backfill
• federation_client.backfill
• transport_layer.backfill -> calls /backfill

From the origin server

• FederationBackfillServlet.on_GET: /backfill/(?P<room_id>[^/]*)/?
• on_backfill_request
• get_backfill_events

Where to fetch the missing insertion event?

• _persist_events_and_state_updates -> _persist_events_txn -> _update_metadata_tables_txn (synapse/storage/databases/main/events.py#L1442)
  • Where we handle insertion/chunk ID's in Make historical events discoverable from backfill for servers without any scrollback history (MSC2716) (federation) #10245
  • We don't have access to async/await stuff and probably not best to call federation code from there
• on_receive_pdu synapse/handlers/federation.py#L193

How to fetch the missing insertion event?

• federation_client.backfill (synapse/federation/federation_client.py#L203): Not for fetching a specific event. Only goes off of extremities and goes backwards from there.

---

• _get_events_and_persist: synapse/handlers/federation.py#L1303
• federation_client.get_pdu: synapse/federation/federation_client.py#L241
• federation_client.transport_layer.get_event: synapse/federation/transport/client.py#L76

---

on_receive_pdu
_get_state_after_missing_prev_event
_get_events_and_persist
get_pdu

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

[MadLittleMods]

Woot! All of the major pieces are merged now 🙌 Full-fat federated historical messages!

@erikjohnston Thanks for the tremendous efforts reviewing each of these big pieces and onboarding me onto all of the Synapse/Matrix concepts to be able to accomplish this 🤠❤

[erikjohnston]

Woo! 🎉 Thanks for bearing with all the back and forth! ❤️