Atomic migrations

[robert-zaremba]

Summary

Today, migrations are not always atomic. If a migration fails, we don't have an easy solution to rollback the changes.

Problem Definition

In-place migration are great addition to the SDK. However if not tested carefully, they can cause extensive problems to a node admin.
When running a migration, changes are written to the disk.

• if we add a new store, then it's committed immediately
• if someone will not use cache wrapped store with open and commit phase (eg because it would be too operational memory consuming).

If a migration fails with an operation listed above, we are leaving a node in a corrupted, possibly unmanageable state. The only way to recover is to sync from another healthy node, or restore a backup.

Proposal

We need to find a more friendly mechanism to handle backup.

ADR-40 will make it easy because we use a DB level checkpoints. But that won't be implemented in 0.44.

Few proposals:

1. flag to cosmovisor to copy the DB (that will basically double the disc requirements).
2. add more options to migration
3. filesystem level rollback based on journal
4. implement ADR-40 based checkpoint mechanism in store - that would limit the number of supported databases and would require DB migration for many nodes.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[robert-zaremba]

When thinking more about it, the option (4.) is the best. The option (1) is the easiest.

[robert-zaremba]

@zmanian , @jackzampolin -- I guess in the past this was not a problem because we were always exporting the state (genesis) and it took a time anyway.

[alexanderbez]

You could do (1) and (4), doing (1) first and if you think you could do (4) in a reasonable amount of time. If not, then at least you can fallback on (1).

[aaronc]

I think it is helpful to clarify that they are only non atomic if KVStores are being added, renamed or deleted. That will only happen if modules are added or removed due to the multi store design. If there are no new modules and just migration code, the migrations will be atomic.

[zmanian]

yeah the way this was achieved in the past was via state exports. This is a good and important observation. We definitely need a checkpoint and recovery mechanism.

Can we get some kind of primitive snapshot functionality into cosmovisor before the cosmoshub upgrade?

[alexanderbez]

@zmanian I think the easiest approach for now is for operators to just take a backup of their data directory prior to upgrading.

[zmanian]

As a practical matter this would mean telling people not to use cosmovisor during the upgrade and I've been asked to tell people to use cosmovisor.

[alexanderbez]

I think they could/should still use it. In the case of an error, they'd have to wipe the resulting data directory and restart cosmovisor?

[anilcse]

I think they could/should still use it. In the case of an error, they'd have to wipe the resulting data directory and restart cosmovisor?

Yes, there's no straight way to rollback in case of upgrade failures. So need to wipe off the data and use backup/or resync.

may be, one quick fix we can add to the cosmovisor is, take a backup if the disk space is available or via a ENV setting, defaults to true. If it's set to false, cosmovisor can just try upgrading and in case of failure, node operators can fix it manually later. In other case, where backup is available, in case of failure, cosmovisor will use skip-upgrade and restart with old binary.

[anilcse]

I think they could/should still use it. In the case of an error, they'd have to wipe the resulting data directory and restart cosmovisor?

Yes, there's no straight way to rollback in case of upgrade failures. So need to wipe off the data and use backup/or resync.

may be, one quick fix we can add to the cosmovisor is, take a backup if the disk space is available or via a ENV setting, defaults to true. If it's set to false, cosmovisor can just try upgrading and in case of failure, node operators can fix it manually later. In other case, where backup is available, in case of failure, cosmovisor will use skip-upgrade and restart with old binary.

Or may be simply,
UNSAFE_UPGRADE:

• If true, the cosmovisor will try to upgrade without any backup
• if false (default), cosmovisor will try to take backup and then upgrade. In case of failure while taking backup, it will just halt the process there and won't try the upgrade.

[alexanderbez]

ACK

[aaronc]

@ethanfrey I wonder if you have any insights on some way we might be able to provide a clean rollback?

I do agree though that the safest thing is always to do a backup so having that as the default like @anilcse is suggesting seems wise.

[aaronc]

Also, are we rolling back the panic on empty minimum gas prices which caused the upgrade failures @anilcse @robert-zaremba ? Seems like we should just emit a warning until the next release.

[amaury1093]

Also, are we rolling back the panic on empty minimum gas prices which caused the upgrade failures @anilcse @robert-zaremba ? Seems like we should just emit a warning until the next release.

this just got merged

[robert-zaremba]

Let's start with option 1. - full backup controlled by a flag.
And later we will use snapshot mechanism.

[robert-zaremba]

BTW: some upgrades can take a lot of memory (eg if we were to update all bank managed storage) if we keep the cash wrapper.

[robert-zaremba]

Oh, @aaronc - I've just noticed that you added "Cosmovisor" to the title and label. I think it's not fully related to the cosmovisor, however it will be partially solved with:

• feat: Add backup option for cosmovisor #9652
• docs: adr-40: reduce multistore and make it atomic #9355
• Multi-store Deprecation #9818

And fully solved with ADR-40 snapshots.

[aaronc]

Right, sorry about that.

[tac0turtle]

closing as this is touched on in the current store discussions

[robert-zaremba]

The issue is still present, so I think we should keep it open until resolved.