StateDb does not have PRUNING_MODE

[tugytur]

When upgrading a node from 0.9.21 to 0.9.22 I get the following error on startup:

2022-05-18 16:37:06      KUSAMA FOUNDATION
2022-05-18 16:37:06 ----------------------------
2022-05-18 16:37:06 Parity Polkadot
2022-05-18 16:37:06 ✌️  version 0.9.22-17c7b9594ae
2022-05-18 16:37:06 ❤️  by Parity Technologies <[email protected]>, 2017-2022
2022-05-18 16:37:06 📋 Chain specification: Kusama
2022-05-18 16:37:06 🏷  Node name: staking
2022-05-18 16:37:06 👤 Role: FULL
2022-05-18 16:37:06 💾 Database: RocksDb at /var/blockchain/data/chains/ksmcc3/db/full
2022-05-18 16:37:06 ⛓  Native runtime: kusama-9220 (parity-kusama-0.tx11.au2)
Error:
   0: State Database error: Invalid metadata: An existing StateDb does not have PRUNING_MODE stored in its meta-data
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                 ⋮ 1 frame hidden ⋮
   2: polkadot::main::h6d5b87adf815dfbf
      at <unknown source file>:<unknown line>
   3: std::sys_common::backtrace::__rust_begin_short_backtrace::h2799de2ee6e5e2fb
      at <unknown source file>:<unknown line>
   4: main<unknown>
      at <unknown source file>:<unknown line>
   5: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
   6: _start<unknown>
      at <unknown source file>:<unknown line>
Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering

ExecStart=/usr/local/bin/polkadot \
        --name "staking" \
        --pruning 1000 \
        --no-mdns \
        --public-addr=/ip4/X.X.X.X/tcp/30333 \
        --listen-addr=/ip4/X.X.X.X/tcp/30333 \
        --chain=kusama \
        --database=rocksdb \
        --prometheus-port 9615 \
        --rpc-port 9933 \
        --ws-port 9944 \
        --no-telemetry \
        --keystore-path=/var/blockchain/keystore \
        --base-path=/var/blockchain/data

Also tried again with --unsafe-pruning \, issue is still there

[paulormart]

I have faced the same issue.

My node was previously synced with the flag sync=Warp and after binary migration to v0.9.22 the following error is shown:

May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 WARNING: "--unsafe-pruning" CLI-flag is deprecated and has no effect. In future builds it will be removed, and providing this flag will lead to an error.                                                                                                                                                  
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 ----------------------------                                                                                                                                                                                                                                                                               
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 This chain is not in any way                                                                                                                                                                                                                                                                               
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46       endorsed by the                                                                                                                                                                                                                                                                                      
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46      KUSAMA FOUNDATION                                                                                                                                                                                                                                                                                     
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 ----------------------------                                                                                                                                                                                                                                                                               
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 Parity Polkadot                                                                                                                                                                                                                                                                                            
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 ✌️  version 0.9.22-17c7b9594ae                                                                                                                                                                                                                                                                              
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 ❤️  by Parity Technologies <[email protected]>, 2017-2022                                                                                                                                                                                                                                                     
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 📋 Chain specification: Kusama                                                                                                                                                                                                                                                                             
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 🏷  Node name: turboflakes-1                                                                                                                                                                                                                                                                                
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 👤 Role: AUTHORITY                                                                                                                                                                                                                                                                                         
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 💾 Database: RocksDb at /polkadot/chains/ksmcc3/db/full                                                                                                                                                                                                                                                    
May 18 17:02:47 localhost polkadot[1315197]: 2022-05-18 17:02:46 ⛓  Native runtime: kusama-9220 (parity-kusama-0.tx11.au2)                                                                                                                                                                                                                                                  
May 18 17:02:48 localhost polkadot[1315197]: Error:                                                                                                                                                                                                                                                                                                                         
May 18 17:02:48 localhost polkadot[1315197]:    0: State Database error: Invalid metadata: An existing StateDb does not have PRUNING_MODE stored in its meta-data                                                                                                                                                                                                           
May 18 17:02:48 localhost polkadot[1315197]: Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.                                                                                                                                                                                                                                               
May 18 17:02:48 localhost polkadot[1315197]: Run with RUST_BACKTRACE=full to include source snippets.                                                                                                                                                                                                                                                                       
May 18 17:02:48 localhost systemd[1]: kusama-node.service: Main process exited, code=exited, status=1/FAILURE                                                                                                                                                                                                                                                               
May 18 17:02:48 localhost systemd[1]: kusama-node.service: Failed with result 'exit-code'.                                                                                                                                                                                                                                                                                  
May 18 17:03:21 localhost systemd[1]: Stopped kusama validator. 

Reverting binary to previous version v0.9.21 the node starts as normal.

[bkchr]

Sadly there was a bug before and the code is now more strict. You would need to resync with warp sync on 0.9.22. Sorry for the inconvenience!

[bkchr]

Okay, there is a problem with warp sync as well: paritytech/polkadot#5556

[tugytur]

Okay, there is a problem with warp sync as well: paritytech/polkadot#5556

I started syncs from scratch for paritydb and rocksdb.
Will make them available on https://parasnaps.io/ as soon as it's synced.

[tugytur]

@bkchr we are syncing since yesterday for Polkadot and Kusama.

Kusama seems a bit slower, but Polkadot is way slower and is syncing only with ~8-15 bps.
Don't know if this is a bug or something drastically changed.

The specs of the VMs are the same as for the Validators (also pass the benchmark tests).

[bkchr]

@tugytur What parameters do you run your node with?

[tugytur]

@bkchr the only differences are the chain and db.

ExecStart=/usr/local/bin/polkadot \
        --name "X" \
        --pruning 1000 \
        --no-mdns \
        --public-addr=/ip4/X.X.X.X/tcp/30333 \
        --listen-addr=/ip4/X.X.X.X/tcp/30333 \
        --chain=XXXX \
        --database=XXXX \
        --prometheus-port 9615 \
        --rpc-port 9933 \
        --ws-port 9944 \
        --no-telemetry \
        --base-path=/var/blockchain

Kusama with paritydb is synced after almost 48h
Kusama with rocksdb after ~50h is at target=#12771417 (50 peers), best: #12755968

Polkadot with paritydb after ~50h target=#10391605 (50 peers), best: #2350890
Polkadot with rocksdb after ~50h target=#10391611 (50 peers), best: #2938044

[bkchr]

This is fixed.