Fix DB serialization order for Ord types

[Leo-Besancon]

An error prone behaviour in our code is that when serializing numbers (or slots) for the keys of the DB, we use varint serializer. As a result, the ordering of such numbers is not garantied in the DB's ordering (for instance when the number is around a power of two).

This is e.g. related to the following issue and associated PR: #4101 and #4104

In this PR, we will replace every instance of such serialization (and deserialization).

TODO:

• List every DB key and check whether it is impacted
• Fix serialization of the keys in the DB
• Fix deserialiazation of the keys in the DB (also in the is_db_key_value_valid) check function!)

Potentially impacted files list. For each file, list the potentially impacted keys:

• https://github.com/massalabs/massa/blob/main/massa-async-pool/src/pool.rs @Leo-Besancon

  • recompute_message_info_cache iterates over the DB (https://github.com/massalabs/massa/blob/main/massa-async-pool/src/pool.rs#L228)
  • fetch_message(message_id) iterates over all the keys with the message_id as a prefix.
  • is_key_value_valid(message_id, message) deserializes the message_id key
  • put_entry(&self, message_id: &AsyncMessageId, message: AsyncMessage, batch: &mut DBBatch) serializes the message_id
  • Same for update_entry(&self, message_id: &AsyncMessageId, message_update: AsyncMessageUpdate, batch: &mut DBBatch)
  • Same for delete_entry
  • For a given (async_message_id, async_message), it is serialized in the DB with:
    • ASYNC_POOL_PREFIX + AsyncMessageIdSerializer::serialize(message_id) + MESSAGE_PART_IDENT

• https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs @AurelienFT

  • https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs#L174 => We iter but is not a problem if we don’t have the right order
  • https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs#L181C34-L181C34 => Shouldn't be an issue as we are not caring about the key value.
  • https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs#L303 => We iter but is not a problem if we don’t have the right order
  • https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs#L326 => Shouldn't be an issue as we are not caring about the key value.
  • https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs#L371 => Shouldn't be an issue as we are not caring about the key value.
  • https://github.com/massalabs/massa/blob/main/massa-db-worker/src/massa_db.rs#L706 => Iter Shouldn't be an issue as we are not caring about the key value.

• https://github.com/massalabs/massa/blob/main/massa-executed-ops/src/executed_ops.rs @AurelienFT => => No problem for me as keys contains only prefix + operation Id which doesn’t have a variant and are not meant to be sorted

• https://github.com/massalabs/massa/blob/main/massa-executed-ops/src/executed_denunciations.rs @AurelienFT

  • https://github.com/massalabs/massa/blob/main/massa-executed-ops/src/executed_denunciations.rs#L63 => Iter but shouldn’t be a problem because it iter on the whole content of denunciations
  • Serialisation in order is needed broken but not assumed anywhere in this file so it’s not a breaking problem.

• https://github.com/massalabs/massa/blob/main/massa-final-state/src/final_state.rs @Eitu33

• pos_state.create_new_cycle_from_last: function from PoS, looks like a potential issue but haven’t looked all the way as I was not assigned to that file
  lines are:
  L234
  L303
  L351
  L380

• pos_state.apply_changes_to_batch: takes slot as input and uses its serialized version when extending deferred credits
  side note: other apply_changes functions in finalize taking slot as input are only using it for pruning purposes
  lines are:
  L450

• when checking db validity, iterates over the whole database and deserializes values to check validity, order should have no impact here
  L556-688

• deletes:
  reset values using constant prefixes
  L128-135
  L848-858
  delete a cycle info with prefix, prefix is built with cycle number
  L224
  L280
  L398

• https://github.com/massalabs/massa/blob/main/massa-ledger-worker/src/ledger_db.rs @Eitu33

• key_serializer_db & key_deserializer_db usage:
  takes care only of ledger keys, which isnt the problem here (version byte + addr serialize + key type)
  side note: to see all of them you can just query “serializer_db” the file

• iterators based on prefixes (again only using ledger key formatting)
  L191
  L463
  L490 (test_export)
  L526 (test_export)

• delete_prefix with base prefix input (basically just a clear)
  L214

• https://github.com/massalabs/massa/blob/main/massa-module-cache/src/hd_cache.rs @Leo-Besancon

  • Is not impacted (only Hash::to_bytes as keys)

• https://github.com/massalabs/massa/blob/main/massa-pos-exports/src/pos_final_state.rs @Leo-Besancon

  • Every key in CYCLE_HISTORY_PREFIX starts with the cycle_prefix serialized as a Varint.
    • complete_key
    • rng_seed_key
    • final_state_hash_snapshot_key
    • roll_count_prefix
    • roll_count_key
    • prod_stats_prefix
    • upper_limit_prefix
    • prod_stats_fail_key
    • prod_stats_success_key
  • Deferred_credits starts with the slot serialized as a Varint of period and varint of thread

• https://github.com/massalabs/massa/blob/main/massa-versioning/src/versioning.rs @sydhds

  • Every key have MIP Info serialized with some varint (version & component version)
    • when we init from the db (e.g. from disk) we load everything, store the udpates in a BTreeMap (thus sorting the MIP Info) then call the update_with functions. This can become a problem if we have way too much MIP info stored in db (> 100 000) ? - Should be discussed..

Impacted keys:

• Critical:
  • All the keys that are prefixed by a cycle (pos_final_state.rs)
  • Deferred credits key (contains a slot - pos_final_state.rs)
• Non critical instances that we won't change in this PR (to avoid too many unnecessary changes that could introduce other problems):
  • In the AsyncPool, the keys in the DB are AsyncMessageId that are Ord and partly encoded as Varint. We do not (currently) use this Ord, but may in the future).
  • Executed Denunciations are indexed by DenunciationIndexed that are Ord and partly encoded as Varint.. We do not (currently) use this Ord, but may in the future)
  • MIP Info related keys, but we don't iter on them in the DB, we query all of them at start.

OTHER FACTORS

• make sure network ban times are 1 day
• update version to MAIN.2.0
• update package versions
• add validated Genesis files
  • wait for validation
  • update files
  • update readme

[damip]

For slots that are used in a db key: use Slot::to_bytes_key() and Slot::from_bytes_key()

For integers, use to_be_bytes() and from_be_bytes()

[AurelienFT]

My part is done.

[damip]

My independent analysis (I went through pos state):

https://github.com/massalabs/massa/pull/4620/files

[Eitu33]

final state and ledger db done

[Leo-Besancon]

I checked every change made in https://github.com/massalabs/massa/pull/4620/files , this PR seems to take into account the same ser/deser.

[codecov]

Codecov Report

Attention: 30 lines in your changes are missing coverage. Please review.

Comparison is base (5697687) 71.78% compared to head (d491cac) 71.78%.

❗ Current head d491cac differs from pull request most recent head 9505422. Consider uploading reports for the commit 9505422 to get more accurate results

Files	Patch %	Lines
massa-pos-exports/src/pos_final_state.rs	80.99%	23 Missing ⚠️
massa-execution-worker/src/execution.rs	69.23%	4 Missing ⚠️
...ssa-execution-worker/src/speculative_roll_state.rs	77.77%	2 Missing ⚠️
massa-node/src/main.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4619   +/-   ##
=======================================
  Coverage   71.78%   71.78%           
=======================================
  Files         291      291           
  Lines       51642    51642           
=======================================
  Hits        37069    37069           
  Misses      14573    14573           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[damip]

All those variables on slices are stressing me out a bit.
Here is a less error-prone approach:

fn buf_to_array_ctr<F: Fn(&[u8; N]) -> V, V, const N: usize>(
    buf: &[u8],
    ctr: F,
) -> Option<(&[u8], V)> {
    Some((&buf[N..], ctr(&buf.get(..N)?.try_into().ok()?)))
}

Then you can do:

let (rest_key, slot) =
    buf_to_array_ctr(rest_key, Slot::from_bytes_key).expect("error");

let (rest_key, cycle) =
    buf_to_array_ctr(rest_key, |v| u64::from_be_bytes(*v)).expect("error");

[damip]

Only the genesis files update remains. Waiting for external approval