rbw client fails to access entries with failed to decrypt: failed to decrypt encrypted secret: invalid mac

[polyzen]

Subject of the issue

rbw client fails to access entries with the error failed to decrypt: failed to decrypt encrypted secret: invalid mac:
doy/rbw#163

A user there stated "it may be your database schema didn't get updated on your account. I would suggest going to their project and figure out how to export and reimport fresh."

Deployment environment

Your environment (Generated via diagnostics page)

• Vaultwarden version: v1.31.0
• Web-vault version: v2024.5.1 (actually v2024.5.1b)
• OS/Arch: linux/arm (Arch Linux ARM)
• Running within a container: false (Base: Not applicable)
• Environment settings overridden: false
• Uses a reverse proxy: true
• IP Header check: true (X-Real-IP)
• Internet access: true
• Internet access via a proxy: false
• DNS Check: true
• Browser/Server Time Check: true
• Server/NTP Time Check: true
• Domain Configuration Check: true
• HTTPS Check: true
• Database type: SQLite
• Database version: 3.45.0
• Clients used: web, Android, rbw
• Reverse proxy and version: nginx 1.26.1
• Other relevant information: Worked fine with with these versions up until a few days ago. Perhaps due to a updated package on the clients and/or server? Web and Android clients seemingly unaffected.

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden:

{
  "_duo_akey": null,
  "_enable_duo": true,
  "_enable_email_2fa": false,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": null,
  "allowed_iframe_ancestors": "",
  "attachments_folder": "/var/lib/vaultwarden/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "/var/lib/vaultwarden",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "*******************************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": true,
  "disable_icon_download": false,
  "domain": "*****://**************",
  "domain_origin": "*****://**************",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_2fa_auto_fallback": false,
  "email_2fa_enforce_on_verified_invite": false,
  "email_attempts_limit": 3,
  "email_change_allowed": true,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": true,
  "enable_websocket": true,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "experimental_client_feature_flags": "fido2-vault-credentials",
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "/var/lib/vaultwarden/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": false,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": "/var/log/vaultwarden.log",
  "log_level": "warn",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "****************************",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 600000,
  "push_enabled": false,
  "push_identity_uri": "https://identity.bitwarden.com",
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "/var/lib/vaultwarden/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "/var/lib/vaultwarden/sends",
  "show_password_hint": false,
  "signups_allowed": false,
  "signups_domains_whitelist": "",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "",
  "smtp_from_name": "Vaultwarden",
  "smtp_host": null,
  "smtp_password": null,
  "smtp_port": 587,
  "smtp_security": "starttls",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": null,
  "templates_folder": "/var/lib/vaultwarden/templates",
  "tmp_folder": "/var/lib/vaultwarden/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "user_send_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "/usr/share/webapps/vaultwarden-web",
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

Steps to reproduce

rbw get <entry>

Expected behaviour

Receive the associated password

Actual behaviour

Receive the error:

WARN: failed to decrypt password: failed to decrypt: failed to decrypt encrypted secret: invalid mac
rbw get: couldn't find entry for 'P73 Arch': failed to decrypt: failed to decrypt encrypted secret: invalid mac

Troubleshooting data

Some details may be found here: doy/rbw#163

A fresh account does not receive this error.

[BlackDex]

Did you also used the new mobile beta native client using that account? If so, which device? iOS or Android?

Or which other clients did you used to create new entries?

[polyzen]

I have not tried the new beta yet.

Most likely would've created new entries on either the web or Android client. Not sure if I've actually used rbw to add or edit entries yet. I mostly use rbw get or rofi-rbw to copy/paste stuff.

Edit: I have used rbw remove at least once, but that might've been months ago.

[BlackDex]

And do other clients still work? They show all the ciphers/entries?

[polyzen]

Yes I haven't noticed any issues in the web and Android clients.

[BlackDex]

Well, i do not want to bounce you around, but then i think it must be something in the rbw client.

Best would be if you are able to pinpoint the issue to a specific item in the vault and look in the database if something is off compared to others.

[polyzen]

Well, i do not want to bounce you around, but then i think it must be something in the rbw client.

No worries.

Best would be if you are able to pinpoint the issue to a specific item in there vault and look in the database of something is off compared to others.

😅

[awpk]

I also tried the new app and now I can't use any client.
Desktop -> loads forever after login
Web -> just no entries
new iOS app -> shows this error
old iOS app -> crashes

EDIT: Browser extension just shows a very limited amount of entries

[awpk]

Sorry, overread the error message above a bit. Mine is "Cryptography error, The cipher's MAC doesn't match the expected value.

[BlackDex]

Well, that is a possible known issue with the native iOS client.
We are not yet sure if this is a Vaultwarden issue, or a client issue, i have not yet checked the Bitwarden repo's for this my self.

[awpk]

I'm having a real issue here not having any access, not even cached on the devices I was logged in :/

[BlackDex]

You should restore a backup, or somehow delete the cipher from the database which is causing this issue.

[awpk]

I need my passwords to get access to backups restore 🤦 (yep will change that).
How can I find out which one it is?

[BlackDex]

Probably the last one you edited, and then probably the one you did via the iOS Native client.
So, i would suggest to to the following.

1. Backup the current (broken) database.
2. Open the database, and then specifically the ciphers table.
3. Sort on updated_at so that the newest items are at the top.
4. Verify that it is your item, and delete that row. If you created/edited multiple items, i suggest to remove all of them.

If you need to know your users uuid, check the users table and look for your email address and check the uuid there.

[awpk]

Thank you so much @BlackDex for your fast answers.
This worked perfectly. Needed to relogin on Desktop and reinstall on iOS but I'm back in.
I will go now do som changes to my procedures and emergency measures 😅

Not sure which entries I deleted now but I guess I will find out one day.

[BlackDex]

Hehe, you can at least now access your backup, and maybe even compare the databases and see which entries are removed. You should be able to copy that row over to the active database and it should be there again.

It would also help to determine the possible issue with rbw or Vaultwarden. There might be something in the broken database which might help.

[awpk]

I actually just remember at least one of the entries (deleted 2) and I did some saving the entry and sharing to an organisation at the same time.

The entry also was quite important. So the row I just deleted could be copied back and it should work?
Where is the thing then that broke it?

[BlackDex]

I wouldn't copy back the entry you deleted, at least not from the broken export at least.
But, if you know the uuid of the ciphers, you can always copy those from a backup into the ciphers table.

For me, and probably the rbw devs as well, it would be good to know what the difference is between the working record and the broken record.

It might be something in the data column, but it could be anything. Even a broken encryption of the name, notes or fields entries.

In theory, you should be able to share those entries without any issue, since all is stored encrypted, and without your keys we can't do anything with those items at all. If you want to share it with us, and make it a bit safer you could do the following.

As an example.
The original entry:

2.ggLuZUIIUqRaxSQcpBVcYA==|0MmQGYpdX+WtoQoM1iKOGA==|fzDN2i05MyRt7PZPG4ByYbimK3bcCtN5aofmOoNSJ24=

See there are 3 segments divided by a |. You could convert all those items to something like this.

2.gg...VcYA==|0MmQ...KOGA==|fzDN...SJ24=

Just keep the first 4 and last 4 characters (exclude the = from the count)
Some entries like the data and fields column have multiple of these items.

Original:

{"AutofillOnPageLoad":null,"Password":"2.21/WbpkcdSJZyzXm/gXiIg==|Wk7Edj2n0uQr94zZjZ3yKw==|lToHSjibPuMJEUNq7TgJre8myqMqzVv374ClyKi8jtE=","PasswordRevisionDate":null,"Totp":"2.Qv+JRctNvGfGrOxIvOOY3Q==|rShmdcGEvMUtYDbKpmgwyw==|qpNTmgt8sQgBIUPWX4g2bZHKd6EmRks40kYcHho7guc=","Uris":[{"Match":null,"Uri":"2.9nTXxMkgLvZ6pQo9GoSZkw==|Jyozx3h2bk5raDO/uaVf3g==|BMeHHYweAUeORDFXufKmtVQXWsjdyC4EQ47PK645+5g="}],"Username":"2.17BMes1tmTjPYwr7TsXZjw==|XH1sQjklg8Lr8ae+mXitzg==|ZP2jO9bS7fwaM7rxoN74YCJCTYFNgFwQPiPpER+LQJ4="}

Safer:

{"AutofillOnPageLoad":null,"Password":"2.21...XiIg==|Wk7E...3yKw==|lToH...8jtE=","PasswordRevisionDate":null,"Totp":"2.Qv...OY3Q==|rShm...gwyw==|qpNT...7guc=","Uris":[{"Match":null,"Uri":"2.9n...SZkw==|Jyoz...Vf3g==|BMeH...5+5g="}],"Username":"2.17...XZjw==|XH1s...itzg==|ZP2j...LQJ4="}

That might help use pinpoint the specific issue which caused all the clients to break down.

[awpk]

I wouldn't copy back the entry you deleted, at least not from the broken export at least.
But, if you know the uuid of the ciphers, you can always copy those from a backup into the ciphers table.

Though this only works when updated I guess? I just created the entry yesterday so it will not be in a backup.

Is there a manual way I could decrypt the data to at least get raw data to manually enter back into my working vault as I actually would need at least an information from the "Notes" section of that entry to reset my password there.

I will look at the steps you provided later to hopefully give you some information about it.

[BlackDex]

I just did a quick test, and for me it seems to at least still be accessible via the web-vault.
But you could modify the contents of the broken record per one encrypted item at a time.

You probably do not want to touch the notes column of course, because that is what you need, unless that somehow is what is broken, then you might try to set that to null and keep the rest original.

You can modify the records as follow, one by one.
name > unencrypted_name (This will not work, but it will at least not break)
data > Modify every single entry there to either null or [] one by one. With everything changed it could look like this:

{"autofillOnPageLoad":null,"password":null,"passwordRevisionDate":null,"totp":null,"uris":[],"username":null}

fields > Same as the data column.

And if it all still fails, try to revert all the above and update notes.
notes > NULL (As in SQL NULL, not the text)

The only UI issue would be, that if the username is not able to be decrypted, you can not click on the vault entry.
This can be worked around by clicking the 3dot's at the right of that entry, and click on Clone, that will open the entry and make all the values visible.

Edit:
For every change you make and saved to the database, you need to logout and back in again to make sure it will reload the vault and fetch the updated record.

[polyzen]

I also tried the new app and now I can't use any client. Desktop -> loads forever after login Web -> just no entries new iOS app -> shows this error old iOS app -> crashes

EDIT: Browser extension just shows a very limited amount of entries

Sorry, you seem to have misread. I have not tried the new Bitwarden beta app. You haven't mentioned using rbw, either. I think you're having a different issue? At least the troubleshooting steps seem to be useful for the rbw issue as well 😅

[BlackDex]

@polyzen Still, the same applies for you, we need to know which cipher record is causing the issue, and what the difference is between the record with an issue, and a record without an issue.

If you are able to provide that info, that would be great.

[stefan0xC]

@BlackDex I could be wrong but I think the difference is that Vaultwarden supports cipher key encryption #3990 and rbw does not (yet). At least that seems to be the difference between working entries and not-working ones.

I'm not sure if this can be replicated generally or is specific to my setup because I've been testing the new web-v2024.7.1 but to replicate the issue rbw users are facing I just had to add or change an entry via the web-vault.

OT: Two issues I've encountered with web-v2024.7.1 are that I cannot seem to clone an entry back to my personal vault (web vault complains about a missing key) and I also can't change the user role (because access_all has been removed and is not yet optional).

[BlackDex]

I actually think the key's are not working for us either. I could be wrong also, but haven't seen those being added on new entries actually. Not on 5 or 6.

[stefan0xC]

With web-v2024.5.1 the entries are still decipherable (and also clonable) so I'd consider it functional for Vaultwarden.

Have you checked if the new clients are producing items with cipher key encryption?

[BlackDex]

Ill have to recheck, reverted and replaced some of my test databases, so no clue anymore which had newer entries actually.

I do have a few with keys, but those are older.

[Biepa]

Hey @BlackDex,
This is @phillip-kaiser from another account 😅
I setup a testing instance with the broken db and the result was the same as before. So far so good. I could find the broken cipher. I deleted the two last ciphers in my first test today and the one I thought of is not the cause of this issue as I can see it again after testing out which one of the two is the reason.

So the "broken entry" just has values in "data" and "name".
Now to the suggested steps.

name > unencrypted_name (This will not work, but it will at least not break)

When trying to login after changing the value of the "name" column to "unencrypted_name" this gave a red error message at the top right saying "An Error has occured: Error saving device"
I reverted that step therefore to not mess up with further testing.

data > Modify every single entry there to either null or [] one by one. With everything changed it could look like this:

The "data" column just had "username" in it. Nothing else. I changed that to to be {"username":null} -> can login but still broken.

The only other column filled is "key".

As the db neither worked with the suggested changes to "data" or "name" (assuming I understood everything correctly) here are the values to hopefully help you finding the issue.

name = 2.Z4...pnXw==|kAtx...H3Vw=|jBmm...KCak=
data = {"username":"2.8A...E7VQ==|D5i1...pDpw==|n5i2...5iBw="}
notes, fields, password_history, deleted_at = NULL

Happy to help if there is anything else I can do.

[BlackDex]

Ok, then i think @stefan0xC is correct, and some clients might not be able to handle them, like rbw.

[BlackDex]

So, i quickly checked and it looks like since the v2024.7.x client versions they are now using cipher key encryption by default if the server version is 2024.2.0 or newer.

Since our web-vault does not yet uses this version creating items via that route does not create the special key per cipher.

So, using any client which isn't able to decrypt this will fail.

Therefor I'm going to close this issue as this has nothing to do with Vaultwarden it self, but more with rbw which currently isn't able to decrypt those ciphers. That probably goes for all older clients, not sure which version.

[polyzen]

Thank you all 🙏. I had updated to Bitwarden Mobile 2024.7.0 on the 16th, and now found one entry that I (accidentally?) updated the day of/before I ran into this issue. AFAICT I had re-saved it with no changes.

[BlackDex]

Saving an entry, no matter if you change it or not, will always push the cipher with newly encrypted data i think.