Implement knock feature #6739

Sorunome · 2020-01-18T20:56:01Z

This PR aims to implement the knock feature as proposed in matrix-org/matrix-spec-proposals#2403

Signed-off-by: Sorunome [email protected]

Heads up that I've taken over the implementation of this PR and the MSC now ~ @anoadragon453

While this PR does not currently have tests, I would like to get review on it in the meantime. I've confirmed it works through manual testing with a local and federated room.

Sorry about the size! I've reorganised all of the commits and their contents. Intended to be reviewed commit-by-commit.

Requires: ~~#8751, #8754~~

LINK TO COMMENT CONTAINING CURRENT STATUS OF KNOCKING

TODO:

Return stripped state events from the remote homeserver once knocking has successfully occurred. These will be passed to the client so that they can present some details about the room that the pending knock is on.
Extensions to /sync:
- Stripped state from rooms we've knocked on
- Accepted/rejected come down existing invite or leave fields.
Send accept/reject status of knock over federation
- We can send just send an invite if your knock is accepted
- Can we send a leave event in a transaction when your knock is rejected?
  - After discussion with @richvdh we're going to descope this. This is a difficult problem that's currently affecting disinvites and bans as well, and needs a proper solution, ideally handled separately.
Tests

Extra things:

Add the ability for the user to rescind their knock. This is done by the knocking homeserver changing their membership to leave before they are accepted/rejected.
- This is a little tricky to do over federation as well. We need to know which server to send the leave event though. We could try the homeserver that we originally knocked through, though we don't currently store that information.
- For now we're just rescinding knocks locally

Add knock events to user data exfiltration code:

synapse/synapse/handlers/admin.py

Lines 112 to 117 in ea70f1c

    
           event_id = room.event_id 
        
           invite = await self.store.get_event(event_id, allow_none=True) 
        
           if invite: 
        
               invited_state = invite.unsigned["invite_room_state"] 
        
               writer.write_invite(room_id, invite, invited_state)

A room your knock was rejected from will show up in your Historical rooms section (out of scope; this won't be possible over federation currently).

Sorunome · 2020-01-19T14:20:28Z

Alright, this PR should be ready for review. As the MSC isn't through yet she isn't sure how to proceed, though

richvdh

I've had a brief skim of this, and it looks like a great start, but most of the points I'd make at this stage are better suited to discussion in the MSC. Hence, I'm going to take it out of review until the MSC has seen a bit more attention.

anoadragon453 · 2020-09-29T11:46:17Z

If we send a leave membership event to the a knocking server to inform them that their knock has been denied, the knocking server will need a way to validate that event. This means it'll need the auth chain(?) which that event references. This is preferred to just sending the entire state of the room, as that could leak information.

I also don't think this state could be sent when the knock is initially introduced into the room (during make/send_knock), because what if the state of the room changes between then and when the knock attempt is rejected?

I assume I wouldn't send this information as part of the transaction (which is where I'm hoping to put the leave event), but instead the knocking server could request the room state from the server that sent the leave event. That server would then send over the necessary state (maybe via backfill?), even though the knocking server isn't in the room yet.

Does that sound like a plausible solution to people? Whatever we get working here could also potentially be used to get unbans and disinvites working over federation as well cf. #1563

anoadragon453 · 2020-09-30T17:15:54Z

@richvdh and I had a chat about the above and have concluded that this is a hard problem to solve correctly. It all comes down to the knocking homeserver being able to validate a knock rejection from the room.

While one potential, limited solution is to only be able to have the homeserver that's in the room that delivered the knock be the one that can deliver the rejection, this doesn't have the best UX and will be confusing to users.

A better solution would possibly involve cross-signing, or some other method of validation. Including that into the scope of this project would extend the work by quite a lot. Hence, we've decided to drop it for now, and solve it and disinvites and unbans in a separate piece of work.

synapse/api/room_versions.py

This is a requirement for [knocking](#6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change. For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included. The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event. This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem. In essence this is just moving some inline code to a reusable store method.

anoadragon453 · 2020-11-02T19:00:47Z

Test strategy:

Complement/Sytest

Whole federated flow
- Client A sends a knock
- Server A sends knock to Server B
- Client B gets the knock
- Client A gets state down sync
- Client B invites Client A
- Client A can join the room
Local flow
Knocking requires the new room version
Knocking requires the new join_rule
Knocking over and over is allowed
Rejecting a knock locally

Synapse unit tests

Check that room_knock_state_types config option affects the events that are sent correctly

There's a handy function called maybe_store_room_on_invite which allows us to create an entry in the rooms table for a room and its version for which we aren't joined to yet, but we can reference when ingesting events about. This is currently used for invites where we receive some stripped state about the room and pass it down via /sync to the client, without us being in the room yet. There is a similar requirement for knocking, where we will eventually do the same thing, and need an entry in the rooms table as well. Thus, reusing this function works, however its name needs to be generalised a bit. Separated out from #6739.

`_locally_reject_invite` generates an out-of-band membership event which can be passed to clients, but not other homeservers. This is used when we fail to reject an invite over federation. If this happens, we instead just generate a leave event locally and send it down /sync, allowing clients to reject invites even if we can't reach the remote homeserver. A similar flow needs to be put in place for rescinding knocks. If we're unable to contact any remote server from the room we've tried to knock on, we'd still like to generate and store the leave event locally. Hence the need to reuse, and thus generalise, this method. Separated from #6739.

Hopefully most of these changes are explained through the added comments and error messages. The changes are also described conceptually in the MSC: https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#join-rules

We're ditching the usual idea of having two endpoints for each membership-related endpoint as per the MSC. Thus knocking only gets the more powerful variant (the one that supports room aliases as well as IDs. The reason is also optional. The other small change is just to ensure displaynames get added to the content of this particular membership event.

This table's usefulness is debatable (see matrix-org#9602), but is currently used by both the codebase and tests. Thus for now I'm leaving it in, but it may well be removed in a future PR.

…ring make_knock We may not if we create a knock room when knocking is enabled, then disabling knocking. We don't want to allow knocks in that case.

clokep

LGTM if CI passes.

This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403. Specifically, it: * Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory * Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room Based on #6739. Complement tests for this change: matrix-org/complement#72

) An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules. This PR corrects that by performing the TPAR check *before* persisting the event.

Introduced in #6739

Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.

@Sorunome

Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](#6739), [\#9359](#9359), [\#10167](#10167), [\#10212](#10212), [\#10227](#10227)) - Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](#9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](#10062), [\#10206](#10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](#10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](#10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](#10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](#10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](#10157), [\#10158](#10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](#10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](#10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](#10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](#10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](#10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](#8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](#10122)) - Fix broken links in documentation. ([\#10180](#10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](#10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](#10062), [\#10210](#10210), [\#10238](#10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](#10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](#10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](#9933)) - Add type hints to the federation servlets. ([\#10080](#10080)) - Improve OpenTracing for event persistence. ([\#10134](#10134), [\#10193](#10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](#10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](#10144)) - Refactor EventPersistenceQueue. ([\#10145](#10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](#10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](#10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](#10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](#10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](#10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](#10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](#10183)) - Improve comments in structured logging code. ([\#10188](#10188)) - Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](#10189)) - Remove redundant DNS lookup limiter. ([\#10190](#10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](#10197)) - Expose OpenTracing trace id in response headers. ([\#10199](#10199))

@Sorunome

Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](#6739), [\#9359](#9359), [\#10167](#10167), [\#10212](#10212), [\#10227](#10227)) - Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](#9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](#10062), [\#10206](#10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](#10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](#10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](#10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](#10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](#10157), [\#10158](#10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](#10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](#10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](#10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](#10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](#10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](#8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](#10122)) - Fix broken links in documentation. ([\#10180](#10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](#10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](#10062), [\#10210](#10210), [\#10238](#10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](#10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](#10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](#9933)) - Add type hints to the federation servlets. ([\#10080](#10080)) - Improve OpenTracing for event persistence. ([\#10134](#10134), [\#10193](#10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](#10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](#10144)) - Refactor EventPersistenceQueue. ([\#10145](#10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](#10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](#10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](#10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](#10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](#10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](#10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](#10183)) - Improve comments in structured logging code. ([\#10188](#10188)) - Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](#10189)) - Remove redundant DNS lookup limiter. ([\#10190](#10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](#10197)) - Expose OpenTracing trace id in response headers. ([\#10199](#10199))

@Sorunome

Synapse 1.37.0 (2021-06-29) =========================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Synapse 1.37.0rc1 (2021-06-24) ============================== Features -------- - Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](matrix-org/synapse#6739), [\#9359](matrix-org/synapse#9359), [\#10167](matrix-org/synapse#10167), [\#10212](matrix-org/synapse#10212), [\#10227](matrix-org/synapse#10227)) - Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](matrix-org/synapse#9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](matrix-org/synapse#10062), [\#10206](matrix-org/synapse#10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](matrix-org/synapse#10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](matrix-org/synapse#10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](matrix-org/synapse#10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](matrix-org/synapse#10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](matrix-org/synapse#10157), [\#10158](matrix-org/synapse#10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](matrix-org/synapse#10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](matrix-org/synapse#10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](matrix-org/synapse#10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](matrix-org/synapse#10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](matrix-org/synapse#10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](matrix-org/synapse#8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](matrix-org/synapse#10122)) - Fix broken links in documentation. ([\#10180](matrix-org/synapse#10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](matrix-org/synapse#10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](matrix-org/synapse#10062), [\#10210](matrix-org/synapse#10210), [\#10238](matrix-org/synapse#10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](matrix-org/synapse#10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](matrix-org/synapse#10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](matrix-org/synapse#9933)) - Add type hints to the federation servlets. ([\#10080](matrix-org/synapse#10080)) - Improve OpenTracing for event persistence. ([\#10134](matrix-org/synapse#10134), [\#10193](matrix-org/synapse#10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](matrix-org/synapse#10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](matrix-org/synapse#10144)) - Refactor EventPersistenceQueue. ([\#10145](matrix-org/synapse#10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](matrix-org/synapse#10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](matrix-org/synapse#10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](matrix-org/synapse#10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](matrix-org/synapse#10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](matrix-org/synapse#10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](matrix-org/synapse#10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](matrix-org/synapse#10183)) - Improve comments in structured logging code. ([\#10188](matrix-org/synapse#10188)) - Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](matrix-org/synapse#10189)) - Remove redundant DNS lookup limiter. ([\#10190](matrix-org/synapse#10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](matrix-org/synapse#10197)) - Expose OpenTracing trace id in response headers. ([\#10199](matrix-org/synapse#10199)) Synapse 1.36.0 (2021-06-15) =========================== No significant changes. Synapse 1.36.0rc2 (2021-06-11) ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](matrix-org/synapse#10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](matrix-org/synapse#10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](matrix-org/synapse#10165)) Synapse 1.36.0rc1 (2021-06-08) ============================== Features -------- - Add new endpoint `/_matrix/client/r0/rooms/{roomId}/aliases` from Client-Server API r0.6.1 (previously [MSC2432](matrix-org/matrix-spec-proposals#2432)). ([\#9224](matrix-org/synapse#9224)) - Improve performance of incoming federation transactions in large rooms. ([\#9953](matrix-org/synapse#9953), [\#9973](matrix-org/synapse#9973)) - Rewrite logic around verifying JSON object and fetching server keys to be more performant and use less memory. ([\#10035](matrix-org/synapse#10035)) - Add new admin APIs for unprotecting local media from quarantine. Contributed by @dklimpel. ([\#10040](matrix-org/synapse#10040)) - Add new admin APIs to remove media by media ID from quarantine. Contributed by @dklimpel. ([\#10044](matrix-org/synapse#10044)) - Make reason and score parameters optional for reporting content. Implements [MSC2414](matrix-org/matrix-spec-proposals#2414). Contributed by Callum Brown. ([\#10077](matrix-org/synapse#10077)) - Add support for routing more requests to workers. ([\#10084](matrix-org/synapse#10084)) - Report OpenTracing spans for database activity. ([\#10113](matrix-org/synapse#10113), [\#10136](matrix-org/synapse#10136), [\#10141](matrix-org/synapse#10141)) - Significantly reduce memory usage of joining large remote rooms. ([\#10117](matrix-org/synapse#10117)) Bugfixes -------- - Fixed a bug causing replication requests to fail when receiving a lot of events via federation. ([\#10082](matrix-org/synapse#10082)) - Fix a bug in the `force_tracing_for_users` option introduced in Synapse v1.35 which meant that the OpenTracing spans produced were missing most tags. ([\#10092](matrix-org/synapse#10092)) - Fixed a bug that could cause Synapse to stop notifying application services. Contributed by Willem Mulder. ([\#10107](matrix-org/synapse#10107)) - Fix bug where the server would attempt to fetch the same history in the room from a remote server multiple times in parallel. ([\#10116](matrix-org/synapse#10116)) - Fix a bug introduced in Synapse 1.33.0 which caused replication requests to fail when receiving a lot of very large events via federation. ([\#10118](matrix-org/synapse#10118)) - Fix bug when using workers where pagination requests failed if a remote server returned zero events from `/backfill`. Introduced in 1.35.0. ([\#10133](matrix-org/synapse#10133)) Improved Documentation ---------------------- - Clarify security note regarding hosting Synapse on the same domain as other web applications. ([\#9221](matrix-org/synapse#9221)) - Update CAPTCHA documentation to mention turning off the verify origin feature. Contributed by @aaronraimist. ([\#10046](matrix-org/synapse#10046)) - Tweak wording of database recommendation in `INSTALL.md`. Contributed by @aaronraimist. ([\#10057](matrix-org/synapse#10057)) - Add initial infrastructure for rendering Synapse documentation with mdbook. ([\#10086](matrix-org/synapse#10086)) - Convert the remaining Admin API documentation files to markdown. ([\#10089](matrix-org/synapse#10089)) - Make a link in docs use HTTPS. Contributed by @RhnSharma. ([\#10130](matrix-org/synapse#10130)) - Fix broken link in Docker docs. ([\#10132](matrix-org/synapse#10132)) Deprecations and Removals ------------------------- - Remove the experimental `spaces_enabled` flag. The spaces features are always available now. ([\#10063](matrix-org/synapse#10063)) Internal Changes ---------------- - Tell CircleCI to build Docker images from `main` branch. ([\#9906](matrix-org/synapse#9906)) - Simplify naming convention for release branches to only include the major and minor version numbers. ([\#10013](matrix-org/synapse#10013)) - Add `parse_strings_from_args` for parsing an array from query parameters. ([\#10048](matrix-org/synapse#10048), [\#10137](matrix-org/synapse#10137)) - Remove some dead code regarding TLS certificate handling. ([\#10054](matrix-org/synapse#10054)) - Remove redundant, unmaintained `convert_server_keys` script. ([\#10055](matrix-org/synapse#10055)) - Improve the error message printed by synctl when synapse fails to start. ([\#10059](matrix-org/synapse#10059)) - Fix GitHub Actions lint for newsfragments. ([\#10069](matrix-org/synapse#10069)) - Update opentracing to inject the right context into the carrier. ([\#10074](matrix-org/synapse#10074)) - Fix up `BatchingQueue` implementation. ([\#10078](matrix-org/synapse#10078)) - Log method and path when dropping request due to size limit. ([\#10091](matrix-org/synapse#10091)) - In Github Actions workflows, summarize the Sytest results in an easy-to-read format. ([\#10094](matrix-org/synapse#10094)) - Make `/sync` do fewer state resolutions. ([\#10102](matrix-org/synapse#10102)) - Add missing type hints to the admin API servlets. ([\#10105](matrix-org/synapse#10105)) - Improve opentracing annotations for `Notifier`. ([\#10111](matrix-org/synapse#10111)) - Enable Prometheus metrics for the jaeger client library. ([\#10112](matrix-org/synapse#10112)) - Work to improve the responsiveness of `/sync` requests. ([\#10124](matrix-org/synapse#10124)) - OpenTracing: use a consistent name for background processes. ([\#10135](matrix-org/synapse#10135))

@Sorunome

Synapse 1.37.0 (2021-06-29) =========================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Synapse 1.37.0rc1 (2021-06-24) ============================== Features -------- - Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](matrix-org/synapse#6739), [\#9359](matrix-org/synapse#9359), [\#10167](matrix-org/synapse#10167), [\#10212](matrix-org/synapse#10212), [\#10227](matrix-org/synapse#10227)) - Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](matrix-org/synapse#9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](matrix-org/synapse#10062), [\#10206](matrix-org/synapse#10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](matrix-org/synapse#10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](matrix-org/synapse#10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](matrix-org/synapse#10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](matrix-org/synapse#10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](matrix-org/synapse#10157), [\#10158](matrix-org/synapse#10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](matrix-org/synapse#10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](matrix-org/synapse#10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](matrix-org/synapse#10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](matrix-org/synapse#10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](matrix-org/synapse#10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](matrix-org/synapse#8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](matrix-org/synapse#10122)) - Fix broken links in documentation. ([\#10180](matrix-org/synapse#10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](matrix-org/synapse#10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](matrix-org/synapse#10062), [\#10210](matrix-org/synapse#10210), [\#10238](matrix-org/synapse#10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](matrix-org/synapse#10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](matrix-org/synapse#10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](matrix-org/synapse#9933)) - Add type hints to the federation servlets. ([\#10080](matrix-org/synapse#10080)) - Improve OpenTracing for event persistence. ([\#10134](matrix-org/synapse#10134), [\#10193](matrix-org/synapse#10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](matrix-org/synapse#10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](matrix-org/synapse#10144)) - Refactor EventPersistenceQueue. ([\#10145](matrix-org/synapse#10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](matrix-org/synapse#10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](matrix-org/synapse#10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](matrix-org/synapse#10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](matrix-org/synapse#10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](matrix-org/synapse#10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](matrix-org/synapse#10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](matrix-org/synapse#10183)) - Improve comments in structured logging code. ([\#10188](matrix-org/synapse#10188)) - Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](matrix-org/synapse#10189)) - Remove redundant DNS lookup limiter. ([\#10190](matrix-org/synapse#10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](matrix-org/synapse#10197)) - Expose OpenTracing trace id in response headers. ([\#10199](matrix-org/synapse#10199))

Sorunome marked this pull request as ready for review January 19, 2020 14:15

Sorunome mentioned this pull request Jan 19, 2020

MSC2403: Add "knock" feature matrix-org/matrix-spec-proposals#2403

Merged

babolivier requested a review from a team February 7, 2020 13:58

richvdh reviewed Feb 13, 2020

View reviewed changes

richvdh added the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Jul 2, 2020

anoadragon453 self-assigned this Sep 17, 2020

anoadragon453 removed the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Sep 17, 2020

anoadragon453 added p1 labels Sep 30, 2020

clokep reviewed Oct 5, 2020

View reviewed changes

synapse/api/room_versions.py Outdated Show resolved Hide resolved

anoadragon453 mentioned this pull request Oct 27, 2020

Abstract code for stripping room state into a separate method #8671

Merged

anoadragon453 mentioned this pull request Nov 12, 2020

Generalise _locally_reject_invite #8751

Merged

anoadragon453 force-pushed the soru/knock branch from 97ea3b1 to e2f5f92 Compare November 12, 2020 19:06

anoadragon453 requested review from a team and removed request for a team November 12, 2020 19:09

anoadragon453 mentioned this pull request Nov 12, 2020

Generalise _maybe_store_room_on_invite #8754

Merged

anoadragon453 force-pushed the soru/knock branch 2 times, most recently from d40ccdf to b08bd18 Compare November 16, 2020 14:09

anoadragon453 added 4 commits November 16, 2020 15:39

Add new experimental room version for knocking

079e995

Add xyz.amorgan.knock /versions string

f1f5fa7

Update the event auth rules for knocking

b81617e

Hopefully most of these changes are explained through the added comments and error messages. The changes are also described conceptually in the MSC: https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#join-rules

Add knocked_members column back to room_stats_historical

0294248

This table's usefulness is debatable (see matrix-org#9602), but is currently used by both the codebase and tests. Thus for now I'm leaving it in, but it may well be removed in a future PR.

anoadragon453 requested a review from clokep June 9, 2021 15:52

anoadragon453 added 2 commits June 9, 2021 17:35

Nuke remaining coding: utf-8 line

c5788b8

Ensure we check that we currently support the desired room version du…

15c23c4

…ring make_knock We may not if we create a knock room when knocking is enabled, then disabling knocking. We don't want to allow knocks in that case.

clokep approved these changes Jun 9, 2021

View reviewed changes

Don't accidentally use a RoomVersion object where we want a str

da85886

anoadragon453 merged commit d936371 into matrix-org:develop Jun 9, 2021

anoadragon453 added a commit that referenced this pull request Jun 9, 2021

Carry changelog over from #6739

105997f

anoadragon453 mentioned this pull request Jun 10, 2021

room_stats_historical and user_stats_historical appear unused #9602

Closed

This was referenced Jun 18, 2021

Check third party rules before persisting knocks over federation #10212

Merged

Re-introduce "Leave out optional keys from /sync" change #10214

Merged

richvdh mentioned this pull request Jun 22, 2021

slow schema migrations #10226

Closed

erikjohnston added a commit that referenced this pull request Jun 22, 2021

Fix schema delta to not take as long on large servers

1dbd311

Introduced in #6739

erikjohnston mentioned this pull request Jun 22, 2021

Fix schema delta to not take as long on large servers #10227

Merged

erikjohnston added a commit that referenced this pull request Jun 22, 2021

Fix schema delta to not take as long on large servers (#10227)

33701dc

Introduced in #6739

richvdh mentioned this pull request Jun 29, 2021

Lots of errors after upgrade to 1.37 #10271

Closed

grinapo mentioned this pull request Aug 3, 2021

unbans do not propagate to federated servers (SYN-725) #1563

Closed

hex-m mentioned this pull request Aug 12, 2021

Feature request: Request to join community channels which are invite-only. element-hq/element-web#5788

Closed

anoadragon453 mentioned this pull request Oct 19, 2021

Add knock-related data to the admin export-data command #11119

Closed

anoadragon453 mentioned this pull request Nov 17, 2021

The knocked_members column of room_stats_state is nullable while all other columns are not #11365

Closed

turt2live mentioned this pull request Feb 3, 2022

Remove excess condition on knock->leave check #11900

Merged

4 tasks

dklimpel mentioned this pull request Feb 27, 2023

Add support for knocking to workers #15133

Merged

4 tasks

matrixbot mentioned this pull request Dec 21, 2023

Investigate whether try/catch statement is necessary when building list of target hosts for /make_* endpoints element-hq/synapse#8808

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement knock feature #6739

Implement knock feature #6739

Sorunome commented Jan 18, 2020 •

edited by anoadragon453

Loading

Sorunome commented Jan 19, 2020

richvdh left a comment

anoadragon453 commented Sep 29, 2020

anoadragon453 commented Sep 30, 2020

anoadragon453 commented Nov 2, 2020

clokep left a comment

	event_id = room.event_id
	invite = await self.store.get_event(event_id, allow_none=True)
	if invite:
	invited_state = invite.unsigned["invite_room_state"]
	writer.write_invite(room_id, invite, invited_state)

Implement knock feature #6739

Implement knock feature #6739

Conversation

Sorunome commented Jan 18, 2020 • edited by anoadragon453 Loading

Sorunome commented Jan 19, 2020

richvdh left a comment

Choose a reason for hiding this comment

anoadragon453 commented Sep 29, 2020

anoadragon453 commented Sep 30, 2020

anoadragon453 commented Nov 2, 2020

clokep left a comment

Choose a reason for hiding this comment

Sorunome commented Jan 18, 2020 •

edited by anoadragon453

Loading