Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSC3869: Read event relations with the Widget API #3869

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 171 additions & 0 deletions proposals/3869-widgetapi-read-event-relations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
# MSC3869: Read event relations with the Widget API

[MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) specifies a Widget API that
is able to receive events from the client. It supports reading state events from a room and room
events from a room timeline. It is also possible to provide a filter to only receive selected events.

While the existing APIs are a good fit for receiving live updates and getting state events, it has
some limitations regarding room events. The client only provides events that it already loaded until
a request. But for some use cases, the widget needs to have a reliable way to query _all relevant_
events from a room (ex: have a certain type; belong to a certain application defined grouping).

The polls feature ([MSC3381](https://github.com/matrix-org/matrix-spec-proposals/pull/3381)) uses
serverside aggregation of message relationships
([MSC2675](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2675-aggregations-server.md))
to solve this use case. Related events reference each other by a `rel_type` and the client-server API
provides an endpoint to receive server-aggregated relations.

Having this API available to widgets, it could (1) access existing events to e.g. show an enhanced view
of the polls feature or (2) read own events for a custom use case such as a whiteboard application.

## Proposal

We will add a new interface to the widget API to get all relations of an event with a known event id.
We won't introduce new capabilities but instead rely on the capabilities that were introduced by
[MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762).

The following rules apply:

1. If the requested event-id belongs to a event, and the widget didn't receive the respective
`m.receive.event:<event type>` or `m.receive.state_event:<event type>` capability, the widget
rejects the request.
2. The list of related events only include events that the widget has the respective
`m.receive.event:<event type>` or `m.receive.state_event:<event type>` capability for. Other
events are silently ignored.
robintown marked this conversation as resolved.
Show resolved Hide resolved

To trigger the read, widgets will use a new `fromWidget` request with the action `read_relations`
which takes the following shape:

```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "read_relations",
"data": {
"room_id": "!room-id",
"event_id": "$event-id",
"rel_type": "m.reference",
"event_type": "m.room.message",
"limit": 50,
"from": "from_token",
"to": "to_token",
"direction": "b"
}
}
```

Under `data`, all keys are a representation of the
`_matrix/client/v1/rooms/{roomID}/relations/{event_id}[/{rel_type}[/{event_type}]]` API that was
introduced by
([MSC2675](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2675-aggregations-server.md)).
This also includes the paging parameters.

The `event_id` parameter is a required parameter that represents the parent event to be read.

The `room_id` parameter specifies the room to look within. When undefined, the client should look in
the user's currently viewed room.

The `rel_type` parameter specifies the relationship type of child events to search for. If not
defined, all types will be returned.

The `event_type` parameter specifies the type of child events to search for. If not defined, all
types will be returned.

The `limit`, `from`, and `to` parameters work like described in
([MSC2675](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2675-aggregations-server.md)).

The `direction` parameter is used to specify the direction to search for relations. It has the same
semantic as defined by ([MSC2675](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2675-aggregations-server.md)).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm failing to find the direction parameter in either MSC2675 or the specification.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I mixed up the MSC numbers. I meant MSC3715.

It is available as an experimental option in synapse:

https://github.com/matrix-org/synapse/blob/3dd175b628bab5638165f20de9eade36a4e88147/synapse/rest/client/relations.py#L59-L67

And also part of the fetchRelations function in the matrix-js-sdk where I come across it. Though the name of the query parameter doesn't match with the synapse implementation 🤔:

https://github.com/matrix-org/matrix-js-sdk/blob/8502759e3ee3a2b244a442b8dac2b67809d9270e/src/client.ts#L7298

We don't have a real use case for this feature, yet. So we could as well skip it for now. It also seems like the MSC in question isn't final yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, well I guess it's ultimately up to you whether to include the parameter, since MSC3715 is fairly close to FCP

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then let's keep it in and hope that it becomes stable in synapse and get's fixed in the matrix-js-sdk so we can actually use it. It is also useful because then the relations(...) endpoint in the RoomWidgetClient will also be fully functioning.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also part of the fetchRelations function in the matrix-js-sdk where I come across it. Though the name of the query parameter doesn't match with the synapse implementation 🤔:

matrix-org/matrix-js-sdk@8502759/src/client.ts#L7298

This is element-hq/element-web#22501, I believe.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is vector-im/element-web#22501, I believe.

Looks quite like it. So if we are adding it to the widget api, it would probably best to then name it dir. The client can then decide whether to use the unstable or stable parameter.


This is an example of a minimal request to get an event from the current room:

```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "read_relations",
"data": {
"event_id": "$event-id"
}
}
```

The client SHOULD NOT modify the data of the request. The widget is responsible for producing valid
events - the client MUST pass through any errors, such as permission errors, to the widget using the
standard error response in the Widget API.
robintown marked this conversation as resolved.
Show resolved Hide resolved

If the request was successful, the client sends the following response:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The client is obligated to return what the widget requested and also to fetch all new events?
I think that should be made extra clear because that is something which usually is not the case. The client does not need to fulfill any guarantees in the other MSC's

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean that the client could also respond to this with a local cache of the relations? I implied that it should always get it from CS-API because it's also how it is (and was already) implemented in the matrix-js-sdk/Element. But it might be a good idea to clarify it here.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something along the lines of:

Suggested change
If the request was successful, the client sends the following response:
Widgets can rely on the complete relation lists.
The client is responsible to fetch all missing events form the matrix server.
If the request was successful, the client sends the following response:


```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "create_room",
"data": {
"event_id": "$event-id"
},
"response": {
"original_event": {
"type": "m.room.message",
"..."
},
"chunk": [
{
"type": "m.room.message",
"..."
},
{ "..." }
],
"next_batch": "next_batch_token",
"prev_batch": "prev_batch_token"
}
}
```

The `original_event` field contains the event that has the requested event id.

The `chunk` field contains an array of events that are related to the parent event and matches the
filters and the capabilities. This list might include less events than the specified `limit` due to
the filter operations.

The `next_batch` field is a cursor that can be used in the `from` or `to` fields to get the next page
of events. If undefined, there are no more events to receive.

The `prev_batch` field is a cursor that can be used in the `from` or `to` fields to get the previous
page of events. If undefined, there are no more events to receive.

## Potential issues

In an e2ee room, all the events must be decrypted in the client prior to applying the filters or
providing them to the widget. This can take a considerable amount of time. The widget should take
care that it selects a reasonable `limit` to not run into timeouts in the widget transport layer.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should mention that this is the first widget api call that can force the client to trigger new cs-api calls. (all other msc's are phrased in a way, that the client can decide if it does not send the events.)

(I also think it would be nice to be more explicit what the clients responsibilities are. see other comment)


## Alternatives

We could also add features to let the client "scroll up the room", i.e. trigger the pagination for a
room timeline and stick with the original read interfaces. However, this would potentially load a
lot of unrelated events which slows the read process down. In addition, the client must potentially
decrypt all the messages in the room before being able to filter them accordingly. Using the relations
feature, the decryption problem is still present, but the set of events that must be decrypted and
searched is minimized.

The same limitations would apply if we would consider to provide direct access to the
`GET /_matrix/client/v3/rooms/{roomId}/messages` endpoint.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe another alternative could be listed. Just to track the idea, that if the client also is capable of setting filters + use /messages custom event types could be used for a similar optimization. But I am not sure how expensive setting filters is.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you mean here. Filters won't work for e2e encryption because one would need to read and decrypt every message in the room to decide it which is not practical. But we could add a comment to clarify it.


## Security considerations

The same considerations as in [MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762)
apply. The widget will be able to receive the same set of events, but can just use a different
approach to request them.

## Unstable prefix

While this MSC is not present in the spec, clients and widgets should:

- Use `org.matrix.msc3869.` in place of `m.` in all new identifiers of this MSC.
- Use `org.matrix.msc3869.read_relations` in place of `read_relations` for the action type in the
`fromWidget` requests.
- Only call/support the `action`s if an API version of `org.matrix.msc3869` is advertised.