Changing PowerReduction param causing network halt

[anilcse]

Summary of Bug

PowerReduction is now changed into an on-chain param. Updating it via a param change proposal is causing the consensus failure. It's occuring on a validator getting jailed once after the param is updated. Here's are the logs: https://pastebin.com/5RDu2mwH

Version

v0.43-beta1

Steps to Reproduce

• Create a new chain with multiple validators
• Create a param change proposal to change PowerReduction param

{
    "title": "Staking power reduction param change proposal",
    "description": "Update power reduction",
    "changes": [
        {
            "subspace": "staking",
            "key": "PowerReduction",
            "value": "10000000"
        }
    ],
    "deposit": "10000000stake"
}

• Vote and let the proposal succeed
• Bring one of the validatos down and wait for it to jail
• It causes CONSENSUS FAILURE!

cc @sunnya97 @robert-zaremba @clevinson @aaronc

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[amaury1093]

Is someone taking a look at this? I believe it's one of the last pieces blocking a 0.43-RC

[amaury1093]

I managed to reproduce the error in a script on my am/9447-power-red branch (it's macos-specific, and uses gsed, feel free to tweak to make it work on your OS).

git checkout am/9447-power-red
./power_reduction.sh
# Follow instructions
# Will error with consensus failure at around block 100:
"should never retrieve a jailed validator from the power store"

link to logs of the couple of blocks leading to the consensus error.

[alexanderbez]

The PowerReduction was never meant to be an on-chain param, at least not with further modifications and state changes to account for this. I can't think of what needs to be done, but I don't think it's trivial.

Since you're changing the way the state-machine views and treats validator powers, you're also changing the way validators are retrieved from the state. Specifically, the state-machine returns to Tendermint validator set updates every block (or epoch) via ApplyAndReturnValidatorSetUpdates. Since you've changed PowerReduction, the iteration is now not deterministic in a sense. So it would technically be possible to fetch a jailed validator from the store if the power reduction is changed in a certain way.

[amaury1093]

So the action here is to revert #8505? @sunnya97 are you okay with that?

[amaury1093]

I created a PR to revert 8505: #9495.

I also re-opened #8365 to discuss if it's desirable to have this feature for v0.44, in a way that doesn't break consensus.

[robert-zaremba]

Specifically, the state-machine returns to Tendermint validator set updates every block (or epoch) via ApplyAndReturnValidatorSetUpdates. Since you've changed PowerReduction, the iteration is now not deterministic in a sense.

Why it is not deterministic? Since blocks are finalized, big majority of the validators will have the same state. There is no randomness.

[alexanderbez]

@robert-zaremba because you need to go into the x/staking store and change every validator's power (and maybe other indexes) using the new power reduction value. The current x/params and x/gov model doesn't allow for us to execute arbitrary logic on param changes. With the new x/gov and x/param redesign, this will be trivial.