[Feature] Implement etcd data validation

[shreyas-s-rao]

How to categorize this issue?

/kind enhancement

What would you like to be added:
Steward should perform validation of the etcd DB with the ability to detect data corruptions, check etcd DB file locks (and subsequent recovery), detect volume mismatches (and consequent EtcdMember status updation), and returns well-defined, non-overlapping error codes.

Changes to be made from the existing data validation logic in etcd-backup-restore:

• Do not perform revision checks in case of multi-node clusters. While single-node etcds are guaranteed to have an etcd DB revision always greater than the latest snapshot revision from the backup store, this is not the case with multi-node etcds, since an etcd member can be lagging behind due to various reasons (network delays, member updates, member restarts, member restorations, etc). Hence, revision checks, which return success only if the etcd DB revision is greater than the latest snapshot revision from the backup store, should not be performed for multi-node etcd clusters
• Revision checks are still necessary for single-node etcds. If the DB is not corrupt, but lags in revisions as compared to the backup store, then a partial restoration should be performed, as described in [Feature] Implement failure-tolerant etcd restoration #11 , and a full restoration should be avoided
• Additional revision check upon a WAL flush, should not be performed for multi-node etcd clusters, since it does not provide any additional benefit to the data validation flow
• Explore the usage of already available data validation/verification checks provided by upstream etcd, and keep steward code as lean as possible
• Explore the usage of data corruption alarm to determine data validity; re-use [Feature] Handle etcd alarms #6 if necessary

Why is this needed:
Part of #1