From 96213b58a52ec699bc26233c35c0112533a8ead5 Mon Sep 17 00:00:00 2001 From: Will Hunt Date: Fri, 15 Feb 2019 12:34:19 +0000 Subject: [PATCH 01/27] Initial proposal commit --- proposals/1888-appservice-edus.md | 113 ++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 proposals/1888-appservice-edus.md diff --git a/proposals/1888-appservice-edus.md b/proposals/1888-appservice-edus.md new file mode 100644 index 00000000000..9cc56129db3 --- /dev/null +++ b/proposals/1888-appservice-edus.md @@ -0,0 +1,113 @@ +# MSC1888: Proposal to send EDUs to appservices + +*Note: Text written in italics represents notes about the section or proposal process. This document +serves as an example of what a proposal could look like (in this case, a proposal to have a template) +and should be used where possible.* + +*In this first section, be sure to cover your problem and a broad overview of the solution. Covering +related details, such as the expected impact, can also be a good idea. The example in this document +says that we're missing a template and that things are confusing and goes on to say the solution is +a template. There's no major expected impact in this proposal, so it doesn't list one. If your proposal +was more invasive (such as proposing a change to how servers discover each other) then that would be +a good thing to list here.* + +*If you're having troubles coming up with a description, a good question to ask is "how +does this proposal improve Matrix?" - the answer could reveal a small impact, and that is okay.* + +There can never be enough templates in the world, and MSCs shouldn't be any different. The level +of detail expected of proposals can be unclear - this is what this example proposal (which doubles +as a template itself) aims to resolve. + + +## Proposal + +*Here is where you'll reinforce your position from the introduction in more detail, as well as cover +the technical points of your proposal. Including rationale for your proposed solution and detailing +why parts are important helps reviewers understand the problem at hand. Not including enough detail +can result in people guessing, leading to confusing arguments in the comments section. The example +here covers why templates are important again, giving a stronger argument as to why we should have +a template. Afterwards, it goes on to cover the specifics of what the template could look like.* + +Having a default template that everyone can use is important. Without a template, proposals would be +all over the place and the minimum amount of detail may be left out. Introducing a template to the +proposal process helps ensure that some amount of consistency is present across multiple proposals, +even if each author decides to abandon the template. + +The default template should be a markdown document because the MSC process requires authors to write +a proposal in markdown. Using other formats wouldn't make much sense because that would prevent authors +from copy/pasting the template. + +The template should have the following sections: + +* **Introduction** - This should cover the primary problem and broad description of the solution. +* **Proposal** - The gory details of the proposal. +* **Tradeoffs** - Any items of the proposal that are less desirable should be listed here. Alternative + solutions to the same problem could also be listed here. +* **Potential issues** - This is where problems with the proposal would be listed, such as changes + that are not backwards compatible. +* **Security considerations** - Discussion of what steps were taken to avoid security issues in the + future and any potential risks in the proposal. +* **Conclusion** - A repeat of the problem and solution. + +Furthermore, the template should not be required to be followed. However it is strongly recommended to +maintain some sense of consistency between proposals. + + +## Tradeoffs + +*This is where alternative solutions could be listed. There's almost always another way to do things +and this section gives you the opportunity to highlight why those ways are not as desirable. The +argument made in this example is that all of the text provided by the template could be integrated +into the proposals introduction, although with some risk of losing clarity.* + +Instead of adding a template to the repository, the assistance it provides could be integrated into +the proposal process itself. There is an argument to be had that the proposal process should be as +descriptive as possible, although having even more detail in the proposals introduction could lead to +some confusion or lack of understanding. Not to mention if the document is too large then potential +authors could be scared off as the process suddenly looks a lot more complicated than it is. For those +reasons, this proposal does not consider integrating the template in the proposals introduction a good +idea. + + +## Potential issues + +*Not all proposals are perfect. Sometimes there's a known disadvantage to implementing the proposal, +and they should be documented here. There should be some explanation for why the disadvantage is +acceptable, however - just like in this example.* + +Someone is going to have to spend the time to figure out what the template should actually have in it. +It could be a document with just a few headers or a supplementary document to the process explanation, +however more detail should be included. A template that actually proposes something should be considered +because it not only gives an opportunity to show what a basic proposal looks like, it also means that +explanations for each section can be described. Spending the time to work out the content of the template +is beneficial and not considered a significant problem because it will lead to a document that everyone +can follow. + + +## Security considerations + +*Some proposals may have some security aspect to them that was addressed in the proposed solution. This +section is a great place to outline some of the security-sensitive components of your proposal, such as +why a particular approach was (or wasn't) taken. The example here is a bit of a stretch and unlikely to +actually be worthwhile of including in a proposal, but it is generally a good idea to list these kinds +of concerns where possible.* + +By having a template available, people would know what the desired detail for a proposal is. This is not +considered a risk because it is important that people understand the proposal process from start to end. + + +## Conclusion + +*Repeating the problem and solution in different words helps reviewers understand the problem a bit more. +This section should wrap up any loose ends left in the document, as well as cover a brief overview of the +content in each section. Note that the example here doesn't touch on the specific implementation details +described in the "Proposal" section - just the high-level points made there.* + +Not having a template for people to follow when making their proposals could lead to large differences +between each MSC. This would make it difficult for reviewers, and there's a potential that some information +could be left out by accident. A template written in the same format the proposal process requires would +give authors the ability to understand how to better explain their own proposal. + +A descriptive template would help potential authors comprehend what the proposal process requires by +demonstrating what is expected of a proposal. Although this is more effort up front, it would lead to more +time saved in the future due to questions about the process. From c70ba2befd530588f1fa7c492742fe5828c710e2 Mon Sep 17 00:00:00 2001 From: Will Hunt Date: Fri, 15 Feb 2019 12:54:40 +0000 Subject: [PATCH 02/27] Add body of proposal --- proposals/1888-appservice-edus.md | 184 +++++++++++++++--------------- 1 file changed, 89 insertions(+), 95 deletions(-) diff --git a/proposals/1888-appservice-edus.md b/proposals/1888-appservice-edus.md index 9cc56129db3..868ce2592b7 100644 --- a/proposals/1888-appservice-edus.md +++ b/proposals/1888-appservice-edus.md @@ -1,113 +1,107 @@ -# MSC1888: Proposal to send EDUs to appservices +# MSC1888: Proposal to send EDUs to appservices -*Note: Text written in italics represents notes about the section or proposal process. This document -serves as an example of what a proposal could look like (in this case, a proposal to have a template) -and should be used where possible.* +*Note: This proposal is not ready for review yet and merely is a brief note on what I propose to do.* -*In this first section, be sure to cover your problem and a broad overview of the solution. Covering -related details, such as the expected impact, can also be a good idea. The example in this document -says that we're missing a template and that things are confusing and goes on to say the solution is -a template. There's no major expected impact in this proposal, so it doesn't list one. If your proposal -was more invasive (such as proposing a change to how servers discover each other) then that would be -a good thing to list here.* - -*If you're having troubles coming up with a description, a good question to ask is "how -does this proposal improve Matrix?" - the answer could reveal a small impact, and that is okay.* - -There can never be enough templates in the world, and MSCs shouldn't be any different. The level -of detail expected of proposals can be unclear - this is what this example proposal (which doubles -as a template itself) aims to resolve. +The appservice /transaction API currently supports pushing PDU events (regular message and state events) +however it doesn't provison for EDU events (typing, presence and more). This means that bridges cannot +react to Matrix users who send any typing or presence information in a room the service is part of. +There is an interest amongst the community to have equal bridging on both sides of a bridge, so that +read reciepts and typing notifications can be seen on the remote side. To that end, this proposal +specifiys how these can be pushed to an appservice. ## Proposal -*Here is where you'll reinforce your position from the introduction in more detail, as well as cover -the technical points of your proposal. Including rationale for your proposed solution and detailing -why parts are important helps reviewers understand the problem at hand. Not including enough detail -can result in people guessing, leading to confusing arguments in the comments section. The example -here covers why templates are important again, giving a stronger argument as to why we should have -a template. Afterwards, it goes on to cover the specifics of what the template could look like.* - -Having a default template that everyone can use is important. Without a template, proposals would be -all over the place and the minimum amount of detail may be left out. Introducing a template to the -proposal process helps ensure that some amount of consistency is present across multiple proposals, -even if each author decides to abandon the template. - -The default template should be a markdown document because the MSC process requires authors to write -a proposal in markdown. Using other formats wouldn't make much sense because that would prevent authors -from copy/pasting the template. - -The template should have the following sections: - -* **Introduction** - This should cover the primary problem and broad description of the solution. -* **Proposal** - The gory details of the proposal. -* **Tradeoffs** - Any items of the proposal that are less desirable should be listed here. Alternative - solutions to the same problem could also be listed here. -* **Potential issues** - This is where problems with the proposal would be listed, such as changes - that are not backwards compatible. -* **Security considerations** - Discussion of what steps were taken to avoid security issues in the - future and any potential risks in the proposal. -* **Conclusion** - A repeat of the problem and solution. - -Furthermore, the template should not be required to be followed. However it is strongly recommended to -maintain some sense of consistency between proposals. - +### Changes to the /transaction/ API + +The `PUT /_matrix/app/v1/transactions/{txnId}` API currently supports sending PDUs +via the `events` array. + +```javascript +{ + "events": [ + { + "content": { + "membership": "join", + "avatar_url": "mxc://domain.com/SEsfnsuifSDFSSEF#auto", + "displayname": "Alice Margatroid" + }, + "type": "m.room.member", + "event_id": "$143273582443PhrSn:domain.com", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "sender": "@example:domain.com", + "origin_server_ts": 1432735824653, + "unsigned": { + "age": 1234 + }, + "state_key": "@alice:domain.com" + } + ] +} +``` + +This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to include +a new key `ephemeral` to match the CS APIs `/sync`. + +*Note: Does m.typing bath user_ids together like that? Looks strange* + +```javascript +{ + "ephemeral": [ + { + "type": "m.typing", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "content": { + "user_ids": [ + "@alice:example.com" + ] + } + }, + { + "type": "m.receipt", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "content": { + "$1435641916114394fHBLK:matrix.org": { + "m.read": { + "@rikj:jki.re": { + "ts": 1436451550453 + } + } + } + } + } + ], + "events": [ + // ... + ] +} +``` + +The reason for a new key rather than bundling the events into `events` is that +old implementations may mistake them for PDUs and could cause undefined behaviour. +While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. + +### Expectations of when an EDU should be pushed to an appservice + +It is not clear at face value what should be pushed to an appservice. An appservice +registers interests in rooms and (usually) it's own users, however EDU events are not +tied to a single room in all situtations and as such there needs to be a specified way of +forwarding these events. + +*Note: How do we determine when to send an EDU? If the room_id is shared by any AS user?* ## Tradeoffs -*This is where alternative solutions could be listed. There's almost always another way to do things -and this section gives you the opportunity to highlight why those ways are not as desirable. The -argument made in this example is that all of the text provided by the template could be integrated -into the proposals introduction, although with some risk of losing clarity.* - -Instead of adding a template to the repository, the assistance it provides could be integrated into -the proposal process itself. There is an argument to be had that the proposal process should be as -descriptive as possible, although having even more detail in the proposals introduction could lead to -some confusion or lack of understanding. Not to mention if the document is too large then potential -authors could be scared off as the process suddenly looks a lot more complicated than it is. For those -reasons, this proposal does not consider integrating the template in the proposals introduction a good -idea. - +TBD, but probably some. ## Potential issues -*Not all proposals are perfect. Sometimes there's a known disadvantage to implementing the proposal, -and they should be documented here. There should be some explanation for why the disadvantage is -acceptable, however - just like in this example.* - -Someone is going to have to spend the time to figure out what the template should actually have in it. -It could be a document with just a few headers or a supplementary document to the process explanation, -however more detail should be included. A template that actually proposes something should be considered -because it not only gives an opportunity to show what a basic proposal looks like, it also means that -explanations for each section can be described. Spending the time to work out the content of the template -is beneficial and not considered a significant problem because it will lead to a document that everyone -can follow. - +TBD, but probably some. ## Security considerations -*Some proposals may have some security aspect to them that was addressed in the proposed solution. This -section is a great place to outline some of the security-sensitive components of your proposal, such as -why a particular approach was (or wasn't) taken. The example here is a bit of a stretch and unlikely to -actually be worthwhile of including in a proposal, but it is generally a good idea to list these kinds -of concerns where possible.* - -By having a template available, people would know what the desired detail for a proposal is. This is not -considered a risk because it is important that people understand the proposal process from start to end. - +TBD, but probably some. ## Conclusion -*Repeating the problem and solution in different words helps reviewers understand the problem a bit more. -This section should wrap up any loose ends left in the document, as well as cover a brief overview of the -content in each section. Note that the example here doesn't touch on the specific implementation details -described in the "Proposal" section - just the high-level points made there.* - -Not having a template for people to follow when making their proposals could lead to large differences -between each MSC. This would make it difficult for reviewers, and there's a potential that some information -could be left out by accident. A template written in the same format the proposal process requires would -give authors the ability to understand how to better explain their own proposal. - -A descriptive template would help potential authors comprehend what the proposal process requires by -demonstrating what is expected of a proposal. Although this is more effort up front, it would lead to more -time saved in the future due to questions about the process. +TBD, and there definiely will be one. From 771cafeb0d5b8f14418910145bde5437db162498 Mon Sep 17 00:00:00 2001 From: Sorunome Date: Tue, 14 Jan 2020 17:06:27 +0100 Subject: [PATCH 03/27] rename file --- proposals/{1888-appservice-edus.md => 2409-appservice-edus.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{1888-appservice-edus.md => 2409-appservice-edus.md} (100%) diff --git a/proposals/1888-appservice-edus.md b/proposals/2409-appservice-edus.md similarity index 100% rename from proposals/1888-appservice-edus.md rename to proposals/2409-appservice-edus.md From 7912fda73edb825c06144a613b912ceda50a7bda Mon Sep 17 00:00:00 2001 From: Sorunome Date: Tue, 14 Jan 2020 17:14:54 +0100 Subject: [PATCH 04/27] finish up MSC --- proposals/2409-appservice-edus.md | 35 ++++++++++++++----------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 868ce2592b7..92ba1f23017 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -1,6 +1,7 @@ -# MSC1888: Proposal to send EDUs to appservices +# MSC2409: Proposal to send EDUs to appservices -*Note: This proposal is not ready for review yet and merely is a brief note on what I propose to do.* +*Node: This proposal is a continuation of [MSC1888](https://github.com/matrix-org/matrix-doc/pull/1888) +and deprecates that one.* The appservice /transaction API currently supports pushing PDU events (regular message and state events) however it doesn't provison for EDU events (typing, presence and more). This means that bridges cannot @@ -17,7 +18,7 @@ specifiys how these can be pushed to an appservice. The `PUT /_matrix/app/v1/transactions/{txnId}` API currently supports sending PDUs via the `events` array. -```javascript +```json { "events": [ { @@ -40,12 +41,10 @@ via the `events` array. } ``` -This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to include -a new key `ephemeral` to match the CS APIs `/sync`. +This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to include a new key +`ephemeral` to match the CS APIs `/sync`. -*Note: Does m.typing bath user_ids together like that? Looks strange* - -```javascript +```json { "ephemeral": [ { @@ -88,20 +87,18 @@ registers interests in rooms and (usually) it's own users, however EDU events ar tied to a single room in all situtations and as such there needs to be a specified way of forwarding these events. -*Note: How do we determine when to send an EDU? If the room_id is shared by any AS user?* - -## Tradeoffs - -TBD, but probably some. +An EDU should be sent to an appservice if the `room_id` is shared by any of the registered appservices +users, if possible. For EDUs where that isn't the case, that is `m.presence`, the EDU should be sent +if the sender is present in a room that is shared by any of the registered appservices users. ## Potential issues -TBD, but probably some. +Determining which EDUs to transmit to the appservice could lead to quite some overhead on the +homeservers side. Additionally, more network traffic is produced, potentially straining the local +network and the appservice more. ## Security considerations -TBD, but probably some. - -## Conclusion - -TBD, and there definiely will be one. +The homeserver needs to accuratley determine which EDUs to send to the appservice, as to not leak +any metadata about users. Particularly `m.presence` could be tricky, as no `room_id` is present in +that EDU. From 9339848e7e1bf7ddf3ee76bcc14d1ef1ca873432 Mon Sep 17 00:00:00 2001 From: Sorunome Date: Tue, 1 Sep 2020 10:38:10 +0200 Subject: [PATCH 05/27] address issues --- proposals/2409-appservice-edus.md | 122 ++++++++++++++++++------------ 1 file changed, 73 insertions(+), 49 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 92ba1f23017..51959ef92fc 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -3,41 +3,65 @@ *Node: This proposal is a continuation of [MSC1888](https://github.com/matrix-org/matrix-doc/pull/1888) and deprecates that one.* -The appservice /transaction API currently supports pushing PDU events (regular message and state events) +The [appservice /transactions API](https://matrix.org/docs/spec/application_service/r0.1.2#put-matrix-app-v1-transactions-txnid) +currently supports pushing PDU events (regular message and state events) however it doesn't provison for EDU events (typing, presence and more). This means that bridges cannot react to Matrix users who send any typing or presence information in a room the service is part of. There is an interest amongst the community to have equal bridging on both sides of a bridge, so that read reciepts and typing notifications can be seen on the remote side. To that end, this proposal -specifiys how these can be pushed to an appservice. +specifies how these can be pushed to an appservice. ## Proposal -### Changes to the /transaction/ API +### Changes to the registration file + +In order that appservices don't get flooded with EDUs, appservices have to opt-in to receive them by +setting `receive_edus` to true. A registration file could look like following: + +```yaml +id: "IRC Bridge" +url: "http://127.0.0.1:1234" +as_token: "30c05ae90a248a4188e620216fa72e349803310ec83e2a77b34fe90be6081f46" +hs_token: "312df522183efd404ec1cd22d2ffa4bbc76a8c1ccf541dd692eef281356bb74e" +sender_localpart: "_irc_bot" +# We want to receive EDUs +receive_edus: true +namespaces: + users: + - exclusive: true + regex: "@_irc_bridge_.*" + aliases: + - exclusive: false + regex: "#_irc_bridge_.*" + rooms: [] +``` + +### Changes to the /transactions/ API The `PUT /_matrix/app/v1/transactions/{txnId}` API currently supports sending PDUs via the `events` array. ```json { - "events": [ - { - "content": { - "membership": "join", - "avatar_url": "mxc://domain.com/SEsfnsuifSDFSSEF#auto", - "displayname": "Alice Margatroid" - }, - "type": "m.room.member", - "event_id": "$143273582443PhrSn:domain.com", - "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", - "sender": "@example:domain.com", - "origin_server_ts": 1432735824653, - "unsigned": { - "age": 1234 - }, - "state_key": "@alice:domain.com" - } - ] + "events": [ + { + "content": { + "membership": "join", + "avatar_url": "mxc://domain.com/SEsfnsuifSDFSSEF#auto", + "displayname": "Alice Margatroid" + }, + "type": "m.room.member", + "event_id": "$143273582443PhrSn:domain.com", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "sender": "@example:domain.com", + "origin_server_ts": 1432735824653, + "unsigned": { + "age": 1234 + }, + "state_key": "@alice:domain.com" + } + ] } ``` @@ -46,45 +70,45 @@ This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to i ```json { - "ephemeral": [ - { - "type": "m.typing", - "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", - "content": { - "user_ids": [ - "@alice:example.com" - ] - } - }, - { - "type": "m.receipt", - "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", - "content": { - "$1435641916114394fHBLK:matrix.org": { - "m.read": { - "@rikj:jki.re": { - "ts": 1436451550453 - } - } + "ephemeral": [ + { + "type": "m.typing", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "content": { + "user_ids": [ + "@alice:example.com" + ] + } + }, + { + "type": "m.receipt", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "content": { + "$1435641916114394fHBLK:matrix.org": { + "m.read": { + "@rikj:jki.re": { + "ts": 1436451550453 } - } + } + } } - ], - "events": [ - // ... - ] + } + ], + "events": [ + // ... + ] } ``` The reason for a new key rather than bundling the events into `events` is that -old implementations may mistake them for PDUs and could cause undefined behaviour. +existing appservices may mistake them for PDUs and could cause undefined behaviour. While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. ### Expectations of when an EDU should be pushed to an appservice It is not clear at face value what should be pushed to an appservice. An appservice registers interests in rooms and (usually) it's own users, however EDU events are not -tied to a single room in all situtations and as such there needs to be a specified way of +tied to a single room in all situations and as such there needs to be a specified way of forwarding these events. An EDU should be sent to an appservice if the `room_id` is shared by any of the registered appservices @@ -95,7 +119,7 @@ if the sender is present in a room that is shared by any of the registered appse Determining which EDUs to transmit to the appservice could lead to quite some overhead on the homeservers side. Additionally, more network traffic is produced, potentially straining the local -network and the appservice more. +network and the appservice more. As such, appservices have to opt-in to receive EDUs. ## Security considerations From 49f2087a5567905233767fdc5bfba8f789e51cf1 Mon Sep 17 00:00:00 2001 From: Sorunome Date: Tue, 13 Oct 2020 13:00:20 +0200 Subject: [PATCH 06/27] change key names and add unstable prefix --- proposals/2409-appservice-edus.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 51959ef92fc..f47655a6489 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -17,7 +17,7 @@ specifies how these can be pushed to an appservice. ### Changes to the registration file In order that appservices don't get flooded with EDUs, appservices have to opt-in to receive them by -setting `receive_edus` to true. A registration file could look like following: +setting `push_ephemeral` to true. A registration file could look like following: ```yaml id: "IRC Bridge" @@ -26,7 +26,7 @@ as_token: "30c05ae90a248a4188e620216fa72e349803310ec83e2a77b34fe90be6081f46" hs_token: "312df522183efd404ec1cd22d2ffa4bbc76a8c1ccf541dd692eef281356bb74e" sender_localpart: "_irc_bot" # We want to receive EDUs -receive_edus: true +push_ephemeral: true namespaces: users: - exclusive: true @@ -115,6 +115,12 @@ An EDU should be sent to an appservice if the `room_id` is shared by any of the users, if possible. For EDUs where that isn't the case, that is `m.presence`, the EDU should be sent if the sender is present in a room that is shared by any of the registered appservices users. +## Unstable prefix + +In the transaction body, instead of `ephemeral`, `de.sorunome.msc2409.ephemeral` is used. + +In the registration file, instead of `push_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. + ## Potential issues Determining which EDUs to transmit to the appservice could lead to quite some overhead on the From fdee029520def0d10ab069682b293f2e967fdb3d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 18 May 2021 16:21:08 -0600 Subject: [PATCH 07/27] Clarifications; to-device handling --- proposals/2409-appservice-edus.md | 46 +++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index f47655a6489..86957592fff 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -104,22 +104,40 @@ The reason for a new key rather than bundling the events into `events` is that existing appservices may mistake them for PDUs and could cause undefined behaviour. While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. -### Expectations of when an EDU should be pushed to an appservice +`to-device` messages are a bit special as they are aimed at a particular user/device ID +combo. These events are annotated by the server with a `to_device_id` and `to_user_id` +field at the top level of the EDU for transport to the appservice: -It is not clear at face value what should be pushed to an appservice. An appservice -registers interests in rooms and (usually) it's own users, however EDU events are not -tied to a single room in all situations and as such there needs to be a specified way of -forwarding these events. +```json5 +{ + "type": "m.new_device", + "sender": "@alice:example.com", + "to_user_id": "@_irc_bob:example.org", + "to_device_id": "ABCDEF123", + "content": { + "device_id": "XYZABCDE", + "rooms": ["!726s6s6q:example.com"] + } +} +``` -An EDU should be sent to an appservice if the `room_id` is shared by any of the registered appservices -users, if possible. For EDUs where that isn't the case, that is `m.presence`, the EDU should be sent -if the sender is present in a room that is shared by any of the registered appservices users. +Note that the EDU is otherwise formatted as it would for client-server API transport. -## Unstable prefix +### Expectations of when an EDU should be pushed to an appservice -In the transaction body, instead of `ephemeral`, `de.sorunome.msc2409.ephemeral` is used. +It is not clear at face value what should be pushed to an appservice. Appservices claim +namespaces of users which registers "interest" in the rooms where those users reside, as +well as claiming namespaces of rooms for explicit interest. However, not all EDUs are +associated with a single room (presence, etc). -In the registration file, instead of `push_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. +If the EDU is capable of being associated to a particular room, it should be sent to the +appservice under the same rules as regular events (interest in the room means sending it). +For EDUs which are not associated with a particular room, the appservice receives the EDU +if it contextually *would* apply. For example, a presence update for a user an appservice +shares a room with (or is under the appservice's namespace) would be sent to the appservice. + +To-device messages for devices belonging to the appservice's user namespaces should always +be sent. ## Potential issues @@ -132,3 +150,9 @@ network and the appservice more. As such, appservices have to opt-in to receive The homeserver needs to accuratley determine which EDUs to send to the appservice, as to not leak any metadata about users. Particularly `m.presence` could be tricky, as no `room_id` is present in that EDU. + +## Unstable prefix + +In the transaction body, instead of `ephemeral`, `de.sorunome.msc2409.ephemeral` is used. + +In the registration file, instead of `push_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. From 231084da132af6518ecefa2a42c37937dfe373d2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 18 May 2021 16:54:55 -0600 Subject: [PATCH 08/27] It's not exactly like sync --- proposals/2409-appservice-edus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 86957592fff..07654c2cc38 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -66,7 +66,7 @@ via the `events` array. ``` This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to include a new key -`ephemeral` to match the CS APIs `/sync`. +`ephemeral` to behave similar to the various sections of the CS API `/sync` endpoint. ```json { From 0c794bceec38eb7c307941ef73c06014c5942642 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 23 Nov 2021 21:51:22 -0700 Subject: [PATCH 09/27] Move to-device messages --- proposals/2409-appservice-edus.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 07654c2cc38..c9175858dd3 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -104,24 +104,31 @@ The reason for a new key rather than bundling the events into `events` is that existing appservices may mistake them for PDUs and could cause undefined behaviour. While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. -`to-device` messages are a bit special as they are aimed at a particular user/device ID +Note that the EDU is otherwise formatted as it would for client-server API transport. + +To-device messages are a bit special as they are aimed at a particular user/device ID combo. These events are annotated by the server with a `to_device_id` and `to_user_id` -field at the top level of the EDU for transport to the appservice: +field at the top level of the message for transport to the appservice: ```json5 { - "type": "m.new_device", + "type": "org.example.to_device_event_type", "sender": "@alice:example.com", "to_user_id": "@_irc_bob:example.org", "to_device_id": "ABCDEF123", "content": { - "device_id": "XYZABCDE", - "rooms": ["!726s6s6q:example.com"] + "hello": "world" } } ``` -Note that the EDU is otherwise formatted as it would for client-server API transport. +Unlike other ephemeral events, to-device messages are included at a top level `to_device` +array in the transaction. If there are no messages to be sent, the array can be omitted. +This is primarily due to how to-device messages work over federation: they get wrapped in +an EDU (`m.direct_to_device`) but that parent EDU is stripped out before sending the message +off to clients. This can lead to potential conflict where if down the line we support EDUs +and to-device messages with the same event type: consumers would be uncertain as to whether +they are handling an EDU or to-device message. ### Expectations of when an EDU should be pushed to an appservice From 9d201459a4d8449aa20a048d6830b969691d60d4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 23 Nov 2021 21:53:28 -0700 Subject: [PATCH 10/27] Copy edu_type behaviour --- proposals/2409-appservice-edus.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index c9175858dd3..449f74ef111 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -72,7 +72,7 @@ This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to i { "ephemeral": [ { - "type": "m.typing", + "edu_type": "m.typing", "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", "content": { "user_ids": [ @@ -81,7 +81,7 @@ This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to i } }, { - "type": "m.receipt", + "edu_type": "m.receipt", "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", "content": { "$1435641916114394fHBLK:matrix.org": { From 91436e4352b1f423667cd27c53bdabcdcdb042dd Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 23 Nov 2021 21:57:24 -0700 Subject: [PATCH 11/27] Add full transaction example --- proposals/2409-appservice-edus.md | 47 +++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 449f74ef111..760a9529d11 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -130,6 +130,53 @@ off to clients. This can lead to potential conflict where if down the line we su and to-device messages with the same event type: consumers would be uncertain as to whether they are handling an EDU or to-device message. +A complete example of the transaction with all 3 arrays populated would be: + +```json5 +{ + "ephemeral": [ + { + "edu_type": "m.typing", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "content": { + "user_ids": [ + "@alice:example.com" + ] + } + } + ], + "events": [ + { + "content": { + "membership": "join", + "avatar_url": "mxc://domain.com/SEsfnsuifSDFSSEF#auto", + "displayname": "Alice Margatroid" + }, + "type": "m.room.member", + "event_id": "$143273582443PhrSn:domain.com", + "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", + "sender": "@example:domain.com", + "origin_server_ts": 1432735824653, + "unsigned": { + "age": 1234 + }, + "state_key": "@alice:domain.com" + } + ], + "to_device": [ + { + "type": "org.example.to_device_event_type", + "sender": "@alice:example.com", + "to_user_id": "@_irc_bob:example.org", + "to_device_id": "ABCDEF123", + "content": { + "hello": "world" + } + } + ] +} +``` + ### Expectations of when an EDU should be pushed to an appservice It is not clear at face value what should be pushed to an appservice. Appservices claim From ce393b1b3dffa0cf7a0fdcc6250474873e9212f2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 23 Nov 2021 22:03:58 -0700 Subject: [PATCH 12/27] Add implementation notes for to-device cleanup --- proposals/2409-appservice-edus.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 760a9529d11..046bc6ce41a 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -193,6 +193,27 @@ shares a room with (or is under the appservice's namespace) would be sent to the To-device messages for devices belonging to the appservice's user namespaces should always be sent. +### Implementation detail: when to delete a to-device message + +Not defined by this MSC is an explicit algorithm for when to delete a to-device message (mark +it as sent). This is left as an implementation detail, though a suggested approach is as +follows: + +* If the message is sent to a user under an appservice's *exclusive* namespace, mark it as sent + and delete it. Note that retried transactions will still need to include the message. +* If the message is sent to a user under an appservice's *inclusive* namespace, mark it as sent + to the appservice but do not delete it until a `/sync` request is completed which includes the + message. Note that retried transactions will still need to include the message. + +This approach is largely to align with how namespaces are used by appservices in practice, but +is not applicable to all scenarios (and thus is an implementation detail). The majority of known +appservices use exclusive namespaces, which typically also means that those users will not be +calling `/sync`. Because of this, the server may never get another opportunity to delete the +messages until it has confirmed that the appservice received the transaction successfully. Inclusive +namespaces are typically used when the appservice wants to impact a subset of users, but without +controlling those users explicitly. Typically, inclusive users are also calling `/sync` and so +the appservice should be CC'd on the to-device messages that would normally go down `/sync`. + ## Potential issues Determining which EDUs to transmit to the appservice could lead to quite some overhead on the @@ -209,4 +230,6 @@ that EDU. In the transaction body, instead of `ephemeral`, `de.sorunome.msc2409.ephemeral` is used. +In the transaction body, instead of `to_device`, `de.sorunome.msc2409.to_device` is used. + In the registration file, instead of `push_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. From c21f86a5da77cc0e0e6daea9f5cd3a9670d0d218 Mon Sep 17 00:00:00 2001 From: Will Hunt Date: Fri, 20 Oct 2023 13:27:40 +0100 Subject: [PATCH 13/27] Use type instead of edu_type to match realities of implementations --- proposals/2409-appservice-edus.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 046bc6ce41a..49c32e64286 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -72,7 +72,7 @@ This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to i { "ephemeral": [ { - "edu_type": "m.typing", + "type": "m.typing", "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", "content": { "user_ids": [ @@ -81,7 +81,7 @@ This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to i } }, { - "edu_type": "m.receipt", + "type": "m.receipt", "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", "content": { "$1435641916114394fHBLK:matrix.org": { @@ -136,7 +136,7 @@ A complete example of the transaction with all 3 arrays populated would be: { "ephemeral": [ { - "edu_type": "m.typing", + "type": "m.typing", "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", "content": { "user_ids": [ From 15f4582c9b00072851033e705881ecfea72c04fa Mon Sep 17 00:00:00 2001 From: Will Hunt Date: Wed, 15 Nov 2023 16:32:21 +0000 Subject: [PATCH 14/27] Add note to say ephemeral can be omitted. --- proposals/2409-appservice-edus.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 49c32e64286..7a9df7fe424 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -66,7 +66,8 @@ via the `events` array. ``` This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to include a new key -`ephemeral` to behave similar to the various sections of the CS API `/sync` endpoint. +`ephemeral` to behave similar to the various sections of the CS API `/sync` endpoint. The `ephemeral` key +MAY be omitted entirely if there are ephemeral no events to send. ```json { From d1783c4569b49061bb702df35c8bf7c1f2b7ddb7 Mon Sep 17 00:00:00 2001 From: Will Hunt Date: Wed, 15 Nov 2023 16:33:40 +0000 Subject: [PATCH 15/27] Improve wording on why we use a seperate array. Co-authored-by: Kevin Cox --- proposals/2409-appservice-edus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 7a9df7fe424..71599af54b7 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -102,7 +102,7 @@ MAY be omitted entirely if there are ephemeral no events to send. ``` The reason for a new key rather than bundling the events into `events` is that -existing appservices may mistake them for PDUs and could cause undefined behaviour. +existing appservices may mistake them for PDUs and might behave erratically. While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. Note that the EDU is otherwise formatted as it would for client-server API transport. From f4b1ec813bac8d8f870d81935a1862a7d461340d Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Sat, 28 Sep 2024 23:37:17 +0300 Subject: [PATCH 16/27] push_ephemeral -> receive_ephemeral --- proposals/2409-appservice-edus.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 71599af54b7..2cdf8cef72a 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -17,7 +17,7 @@ specifies how these can be pushed to an appservice. ### Changes to the registration file In order that appservices don't get flooded with EDUs, appservices have to opt-in to receive them by -setting `push_ephemeral` to true. A registration file could look like following: +setting `receive_ephemeral` to true. A registration file could look like following: ```yaml id: "IRC Bridge" @@ -26,7 +26,7 @@ as_token: "30c05ae90a248a4188e620216fa72e349803310ec83e2a77b34fe90be6081f46" hs_token: "312df522183efd404ec1cd22d2ffa4bbc76a8c1ccf541dd692eef281356bb74e" sender_localpart: "_irc_bot" # We want to receive EDUs -push_ephemeral: true +receive_ephemeral: true namespaces: users: - exclusive: true @@ -233,4 +233,4 @@ In the transaction body, instead of `ephemeral`, `de.sorunome.msc2409.ephemeral` In the transaction body, instead of `to_device`, `de.sorunome.msc2409.to_device` is used. -In the registration file, instead of `push_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. +In the registration file, instead of `receive_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. From 3a8fc4d701b9eaebed33f9e84421eb27e44a4834 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Sat, 28 Sep 2024 23:37:48 +0300 Subject: [PATCH 17/27] Fix some typos and clarify EDU room association --- proposals/2409-appservice-edus.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 2cdf8cef72a..d1eaa073556 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -67,7 +67,7 @@ via the `events` array. This proposal would extend the `PUT /_matrix/app/v1/transactions/` endpoint to include a new key `ephemeral` to behave similar to the various sections of the CS API `/sync` endpoint. The `ephemeral` key -MAY be omitted entirely if there are ephemeral no events to send. +MAY be omitted entirely if there are no ephemeral events to send. ```json { @@ -185,7 +185,7 @@ namespaces of users which registers "interest" in the rooms where those users re well as claiming namespaces of rooms for explicit interest. However, not all EDUs are associated with a single room (presence, etc). -If the EDU is capable of being associated to a particular room, it should be sent to the +If the EDU is capable of being associated to a particular room (i.e. `m.typing` and `m.receipt`), it should be sent to the appservice under the same rules as regular events (interest in the room means sending it). For EDUs which are not associated with a particular room, the appservice receives the EDU if it contextually *would* apply. For example, a presence update for a user an appservice @@ -223,7 +223,7 @@ network and the appservice more. As such, appservices have to opt-in to receive ## Security considerations -The homeserver needs to accuratley determine which EDUs to send to the appservice, as to not leak +The homeserver needs to accurately determine which EDUs to send to the appservice, as to not leak any metadata about users. Particularly `m.presence` could be tricky, as no `room_id` is present in that EDU. From 18abe04bf6e13cab2396935056c5931f2d8ea8e8 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Sat, 28 Sep 2024 23:38:41 +0300 Subject: [PATCH 18/27] Clarify EDU formatting --- proposals/2409-appservice-edus.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index d1eaa073556..43750c71527 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -105,7 +105,11 @@ The reason for a new key rather than bundling the events into `events` is that existing appservices may mistake them for PDUs and might behave erratically. While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. -Note that the EDU is otherwise formatted as it would for client-server API transport. +EDUs are formatted the same way as they are in the client-server API `/sync` +endpoint, with the addition of the `room_id` field for room-scoped EDUs +(`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because +sync nests EDUs inside a room object, but appservices get a flat list of events +in all rooms. To-device messages are a bit special as they are aimed at a particular user/device ID combo. These events are annotated by the server with a `to_device_id` and `to_user_id` From 5989fc86446909d23c89ddb65d323bc81889c4db Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Sat, 28 Sep 2024 23:43:42 +0300 Subject: [PATCH 19/27] Explicitly list all event types --- proposals/2409-appservice-edus.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 43750c71527..d612fff6e86 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -5,7 +5,7 @@ and deprecates that one.* The [appservice /transactions API](https://matrix.org/docs/spec/application_service/r0.1.2#put-matrix-app-v1-transactions-txnid) currently supports pushing PDU events (regular message and state events) -however it doesn't provison for EDU events (typing, presence and more). This means that bridges cannot +however it doesn't provison for EDU events (typing, presence and receipts). This means that bridges cannot react to Matrix users who send any typing or presence information in a room the service is part of. There is an interest amongst the community to have equal bridging on both sides of a bridge, so that @@ -105,11 +105,13 @@ The reason for a new key rather than bundling the events into `events` is that existing appservices may mistake them for PDUs and might behave erratically. While `events` may now be a somewhat misleading name, this is an acceptable tradeoff. -EDUs are formatted the same way as they are in the client-server API `/sync` -endpoint, with the addition of the `room_id` field for room-scoped EDUs -(`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because -sync nests EDUs inside a room object, but appservices get a flat list of events -in all rooms. +The array is effectively a combination of the `presence` and `ephemeral` sections of the +client-server `/sync` API. User-defined ephemeral events don't exist yet, which means there are +only three event types that can currently occur: `m.presence`, `m.typing` and `m.receipt`. + +EDUs are formatted the same way as they are in C-S sync, with the addition of the `room_id` field +for room-scoped EDUs (`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because +sync nests EDUs inside a room object, but appservices get a flat list of events in all rooms. To-device messages are a bit special as they are aimed at a particular user/device ID combo. These events are annotated by the server with a `to_device_id` and `to_user_id` From d7fb52af0c029593cd75191849fb6b90e5a6fdeb Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Sat, 28 Sep 2024 23:38:54 +0300 Subject: [PATCH 20/27] Delete to-device events to be moved to a new MSC --- proposals/2409-appservice-edus.md | 97 ------------------------------- 1 file changed, 97 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index d612fff6e86..f28cecba163 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -113,77 +113,6 @@ EDUs are formatted the same way as they are in C-S sync, with the addition of th for room-scoped EDUs (`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because sync nests EDUs inside a room object, but appservices get a flat list of events in all rooms. -To-device messages are a bit special as they are aimed at a particular user/device ID -combo. These events are annotated by the server with a `to_device_id` and `to_user_id` -field at the top level of the message for transport to the appservice: - -```json5 -{ - "type": "org.example.to_device_event_type", - "sender": "@alice:example.com", - "to_user_id": "@_irc_bob:example.org", - "to_device_id": "ABCDEF123", - "content": { - "hello": "world" - } -} -``` - -Unlike other ephemeral events, to-device messages are included at a top level `to_device` -array in the transaction. If there are no messages to be sent, the array can be omitted. -This is primarily due to how to-device messages work over federation: they get wrapped in -an EDU (`m.direct_to_device`) but that parent EDU is stripped out before sending the message -off to clients. This can lead to potential conflict where if down the line we support EDUs -and to-device messages with the same event type: consumers would be uncertain as to whether -they are handling an EDU or to-device message. - -A complete example of the transaction with all 3 arrays populated would be: - -```json5 -{ - "ephemeral": [ - { - "type": "m.typing", - "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", - "content": { - "user_ids": [ - "@alice:example.com" - ] - } - } - ], - "events": [ - { - "content": { - "membership": "join", - "avatar_url": "mxc://domain.com/SEsfnsuifSDFSSEF#auto", - "displayname": "Alice Margatroid" - }, - "type": "m.room.member", - "event_id": "$143273582443PhrSn:domain.com", - "room_id": "!jEsUZKDJdhlrceRyVU:domain.com", - "sender": "@example:domain.com", - "origin_server_ts": 1432735824653, - "unsigned": { - "age": 1234 - }, - "state_key": "@alice:domain.com" - } - ], - "to_device": [ - { - "type": "org.example.to_device_event_type", - "sender": "@alice:example.com", - "to_user_id": "@_irc_bob:example.org", - "to_device_id": "ABCDEF123", - "content": { - "hello": "world" - } - } - ] -} -``` - ### Expectations of when an EDU should be pushed to an appservice It is not clear at face value what should be pushed to an appservice. Appservices claim @@ -197,30 +126,6 @@ For EDUs which are not associated with a particular room, the appservice receive if it contextually *would* apply. For example, a presence update for a user an appservice shares a room with (or is under the appservice's namespace) would be sent to the appservice. -To-device messages for devices belonging to the appservice's user namespaces should always -be sent. - -### Implementation detail: when to delete a to-device message - -Not defined by this MSC is an explicit algorithm for when to delete a to-device message (mark -it as sent). This is left as an implementation detail, though a suggested approach is as -follows: - -* If the message is sent to a user under an appservice's *exclusive* namespace, mark it as sent - and delete it. Note that retried transactions will still need to include the message. -* If the message is sent to a user under an appservice's *inclusive* namespace, mark it as sent - to the appservice but do not delete it until a `/sync` request is completed which includes the - message. Note that retried transactions will still need to include the message. - -This approach is largely to align with how namespaces are used by appservices in practice, but -is not applicable to all scenarios (and thus is an implementation detail). The majority of known -appservices use exclusive namespaces, which typically also means that those users will not be -calling `/sync`. Because of this, the server may never get another opportunity to delete the -messages until it has confirmed that the appservice received the transaction successfully. Inclusive -namespaces are typically used when the appservice wants to impact a subset of users, but without -controlling those users explicitly. Typically, inclusive users are also calling `/sync` and so -the appservice should be CC'd on the to-device messages that would normally go down `/sync`. - ## Potential issues Determining which EDUs to transmit to the appservice could lead to quite some overhead on the @@ -237,6 +142,4 @@ that EDU. In the transaction body, instead of `ephemeral`, `de.sorunome.msc2409.ephemeral` is used. -In the transaction body, instead of `to_device`, `de.sorunome.msc2409.to_device` is used. - In the registration file, instead of `receive_ephemeral`, `de.sorunome.msc2409.push_ephemeral` is used. From 7057ba0d8bf1ab55ab36bfacf30232b9de16dc7a Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Mon, 30 Sep 2024 16:07:23 +0300 Subject: [PATCH 21/27] Update spec link and fix typo Co-authored-by: Patrick Cloke --- proposals/2409-appservice-edus.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index f28cecba163..acb50873eef 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -3,9 +3,9 @@ *Node: This proposal is a continuation of [MSC1888](https://github.com/matrix-org/matrix-doc/pull/1888) and deprecates that one.* -The [appservice /transactions API](https://matrix.org/docs/spec/application_service/r0.1.2#put-matrix-app-v1-transactions-txnid) +The [appservice /transactions API](https://spec.matrix.org/v1.11/application-service-api/#put_matrixappv1transactionstxnid) currently supports pushing PDU events (regular message and state events) -however it doesn't provison for EDU events (typing, presence and receipts). This means that bridges cannot +however it doesn't provision for EDU events (typing, presence and receipts). This means that bridges cannot react to Matrix users who send any typing or presence information in a room the service is part of. There is an interest amongst the community to have equal bridging on both sides of a bridge, so that From 20501b4a70525cff4ec2c828d8123fc6fbdd7898 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Mon, 30 Sep 2024 16:12:10 +0300 Subject: [PATCH 22/27] Add private read receipt rules --- proposals/2409-appservice-edus.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index acb50873eef..95065296df6 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -126,6 +126,9 @@ For EDUs which are not associated with a particular room, the appservice receive if it contextually *would* apply. For example, a presence update for a user an appservice shares a room with (or is under the appservice's namespace) would be sent to the appservice. +For `m.receipt`, private read receipts (`m.read.private`) should only be sent for users within the +appservice's namespaces. Normal read receipts and threaded read receipts are always sent. + ## Potential issues Determining which EDUs to transmit to the appservice could lead to quite some overhead on the From 5a745fd66128ee88580dcedf6c246e127800d71e Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 1 Oct 2024 17:08:18 +0300 Subject: [PATCH 23/27] Apply suggestions from code review Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/2409-appservice-edus.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 95065296df6..ff02a1209d3 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -1,4 +1,4 @@ -# MSC2409: Proposal to send EDUs to appservices +# MSC2409: Proposal to send typing, presence and receipts to appservices *Node: This proposal is a continuation of [MSC1888](https://github.com/matrix-org/matrix-doc/pull/1888) and deprecates that one.* @@ -107,7 +107,7 @@ While `events` may now be a somewhat misleading name, this is an acceptable trad The array is effectively a combination of the `presence` and `ephemeral` sections of the client-server `/sync` API. User-defined ephemeral events don't exist yet, which means there are -only three event types that can currently occur: `m.presence`, `m.typing` and `m.receipt`. +only three event types that can currently occur: [`m.presence`](https://spec.matrix.org/v1.11/client-server-api/#mpresence), [`m.typing`](https://spec.matrix.org/v1.11/client-server-api/#mtyping), and [`m.receipt`](https://spec.matrix.org/v1.11/client-server-api/#mreceipt). EDUs are formatted the same way as they are in C-S sync, with the addition of the `room_id` field for room-scoped EDUs (`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because From fa4d518ddb4381d2cea3a172f852e3f1bb36fc9d Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 1 Oct 2024 17:08:57 +0300 Subject: [PATCH 24/27] Wrap lines --- proposals/2409-appservice-edus.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index ff02a1209d3..0200447f096 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -107,7 +107,10 @@ While `events` may now be a somewhat misleading name, this is an acceptable trad The array is effectively a combination of the `presence` and `ephemeral` sections of the client-server `/sync` API. User-defined ephemeral events don't exist yet, which means there are -only three event types that can currently occur: [`m.presence`](https://spec.matrix.org/v1.11/client-server-api/#mpresence), [`m.typing`](https://spec.matrix.org/v1.11/client-server-api/#mtyping), and [`m.receipt`](https://spec.matrix.org/v1.11/client-server-api/#mreceipt). +only three event types that can currently occur: +[`m.presence`](https://spec.matrix.org/v1.11/client-server-api/#mpresence), +[`m.typing`](https://spec.matrix.org/v1.11/client-server-api/#mtyping), +and [`m.receipt`](https://spec.matrix.org/v1.11/client-server-api/#mreceipt). EDUs are formatted the same way as they are in C-S sync, with the addition of the `room_id` field for room-scoped EDUs (`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because @@ -120,11 +123,12 @@ namespaces of users which registers "interest" in the rooms where those users re well as claiming namespaces of rooms for explicit interest. However, not all EDUs are associated with a single room (presence, etc). -If the EDU is capable of being associated to a particular room (i.e. `m.typing` and `m.receipt`), it should be sent to the -appservice under the same rules as regular events (interest in the room means sending it). -For EDUs which are not associated with a particular room, the appservice receives the EDU -if it contextually *would* apply. For example, a presence update for a user an appservice -shares a room with (or is under the appservice's namespace) would be sent to the appservice. +If the EDU is capable of being associated to a particular room (i.e. `m.typing` and `m.receipt`), +it should be sent to the appservice under the same rules as regular events (interest in the room +means sending it). For EDUs which are not associated with a particular room, the appservice +receives the EDU if it contextually *would* apply. For example, a presence update for a user an +appservice shares a room with (or is under the appservice's namespace) would be sent to the +appservice. For `m.receipt`, private read receipts (`m.read.private`) should only be sent for users within the appservice's namespaces. Normal read receipts and threaded read receipts are always sent. From 543104569af57829a7aabaa6fe13cd9af550c082 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 1 Oct 2024 22:36:46 +0300 Subject: [PATCH 25/27] Apply suggestions from code review Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/2409-appservice-edus.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 0200447f096..b4e4877faf6 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -1,6 +1,6 @@ # MSC2409: Proposal to send typing, presence and receipts to appservices -*Node: This proposal is a continuation of [MSC1888](https://github.com/matrix-org/matrix-doc/pull/1888) +*Note: This proposal is a continuation of [MSC1888](https://github.com/matrix-org/matrix-doc/pull/1888) and deprecates that one.* The [appservice /transactions API](https://spec.matrix.org/v1.11/application-service-api/#put_matrixappv1transactionstxnid) @@ -136,13 +136,13 @@ appservice's namespaces. Normal read receipts and threaded read receipts are alw ## Potential issues Determining which EDUs to transmit to the appservice could lead to quite some overhead on the -homeservers side. Additionally, more network traffic is produced, potentially straining the local +homeserver side. Additionally, more network traffic is produced, potentially straining the local network and the appservice more. As such, appservices have to opt-in to receive EDUs. ## Security considerations The homeserver needs to accurately determine which EDUs to send to the appservice, as to not leak -any metadata about users. Particularly `m.presence` could be tricky, as no `room_id` is present in +any unnecessary metadata about users. Particularly `m.presence` could be tricky, as no `room_id` is present in that EDU. ## Unstable prefix From 842c44e35f6c9b1fa6816f271c8e4cfc4e3f2af4 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 1 Oct 2024 22:37:50 +0300 Subject: [PATCH 26/27] Explicitly mention to-device events are not here --- proposals/2409-appservice-edus.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index b4e4877faf6..89fad81e8ac 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -112,6 +112,9 @@ only three event types that can currently occur: [`m.typing`](https://spec.matrix.org/v1.11/client-server-api/#mtyping), and [`m.receipt`](https://spec.matrix.org/v1.11/client-server-api/#mreceipt). +This proposal does not cover any other types of events which are sent as EDUs in the federation API, +such as to-device events or other e2ee features. Those are left to a separate MSC. + EDUs are formatted the same way as they are in C-S sync, with the addition of the `room_id` field for room-scoped EDUs (`m.typing` and `m.receipt`). `room_id` is not present in the C-S API because sync nests EDUs inside a room object, but appservices get a flat list of events in all rooms. From 94e605e21025ee557be7cce6503698152b80a710 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Fri, 18 Oct 2024 12:19:22 +0300 Subject: [PATCH 27/27] Mention the possibility of more granular filtering --- proposals/2409-appservice-edus.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/2409-appservice-edus.md b/proposals/2409-appservice-edus.md index 89fad81e8ac..2f8effa27be 100644 --- a/proposals/2409-appservice-edus.md +++ b/proposals/2409-appservice-edus.md @@ -37,6 +37,9 @@ namespaces: rooms: [] ``` +For now, receiving EDUs is all-or-nothing. A future MSC may add more granular +filtering capabilities for appservices. + ### Changes to the /transactions/ API The `PUT /_matrix/app/v1/transactions/{txnId}` API currently supports sending PDUs