Megolm Backup | Client should always have a local copy of all their megolm keys to limit the possibility of key loss

[BillCarsonFr]

We do not proactively download message keys from key storage (and instead, download keys on demand after a UTD).

This means that, if the user then resets their key backup (for example because they forgot their recovery key), it is likely that a significant number of message keys will be lost.

To address this, we will make each new client download (in the background) a copy of all the message keys in key storage as soon as it can. The operation should be resilient to app restart and should have a UI to report progress.

We'd rather not implement this twice, so our plan as of this writing is to switch Element Web to using the key-backup management code in matrix-rust-sdk, and then implement this once there.

Tasks:

• Move matrix-sdk::encryption::backups to matrix-sdk-crypto so that it can be used by clients using that crate matrix-org/matrix-rust-sdk#3611
• rust-sdk: Ensure that all the keys from a given backup version are downloaded
• EIX: UI feedback for ongoing import
• EAX: UI feedback for ongoing import
• Key Backup not restored automatically after verification element-web#27009

Previous description

Context

We implemented a recent change to the way we get the room keys from backup. Previously, the full backup was downloaded after verification, as soon as the backup decryption key was known, but that did not result in a good user experience when your backup was too big. The time to download the full backup and decrypt it could be taking 10s of minutes depending on the platform.

In order to improve that (and in the absence of pagination for the backup), keys are now downloaded on-demand when we try to decrypt an historical message.
This results in a better user experience, as the messages get decrypted faster after a new login.

Problem

With this new way of getting the keys, all user sessions will now have only a partial set of the backed up keys. This can cause key loss if for example a user tries to reset his backup (without importing first from the previous backup).

Notice that the previous implementation of full backup import has several issues:

• Performance
• Lack of pause/stop/resume. Particularly on mobile, if the app was put in background and terminated by the OS before the end of import, it was not restarted.
• Was possible to start several import in parallel

Solution

We want to ensure that each new client gets a copy of all the keys in backup as soon as it can. The operation should be resilient to app restart and should have an API [richvdh: or, more to the point: a UI?] to report progress.

• EAX/EIX:
  • rust-sdk: Ensure that all the keys from a given backup version are downloaded
  • EIX: UI feedback for ongoing import
  • EAX: UI feedback for ongoing import

• Web (Key Backup not restored automatically after verification element-web#27009):
  • js-sdk: Ensure that all the keys from a given backup version are downloaded
  • UI should give feedback on key backup restore element-web#17839
• Element Android:
• Element iOS

Related/Existing issues

• Key Backup not restored automatically after verification element-web#27009
• Restore keys (from secure online backup) in background rather than foreground element-web#20694

[richvdh]

I wrote some notes on how we might do this for web at element-hq/element-web#27009.

Notes:

• In a naive implementation, if the app is killed during the recovery, we'll restart from the beginning next time. Anything else is somewhat annoying without a fix to Key backup restore API lacks sensible pagination #2176.
• If your backup is too big, it's possible that we'll never successfully download it; the client will just keep retrying.

[richvdh]

Question: is there any value in attempting to download individual keys on-demand once the full backup has been successfully downloaded? I'd think not?

[andybalaam]

@richvdh will remodel this item based on https://docs.google.com/document/d/1eEj_BagixDuny_fhF_WjoKEm0SD538mTXA3s436-c2I/edit - we chose the "Update Rust backup manager, and refactor EW to use it" option

[BillCarsonFr]

Question: is there any value in attempting to download individual keys on-demand once the full backup has been successfully downloaded? I'd think not?

Short answer: Once it has been successfully downloaded, no. But while it's beeing downoaladed definitly yes.
If not it will take ages before I can see history after a login (unless we have that).

Longer answer:

• Maybe a quick race where when you start downloading, some new keys are beeing uploaded and you'd want to get them after?
• Maybe in case of identity reset some devices might become unsigned for some time and then be excluded from key distribution, but when they are verified again they could re-try the backup?