Add server level feature gate

[siyuanfoundation]

What would you like to be added?

Add the Kubernetes style feature gate framework into etcd to gate future feature enhancement.
Users would be able to turn features on or off for the single server and query feature enablement in a consistent way.

This issue is for tracking KEP-4578.
Tasks in this KEP include

• Milestone 1

• server feature gate interface defined, and code copied from K8s into pkg/: featuregate: adds EtcdServer.FeatureEnabled interface. #18062.
• --feature-gates flag added: Add "server-feature-gates" flag. #18279, Add config file field for feature-gates flag. #18365.
• feature gate added to the server code, and used by a server level experimental feature: migrate experimental-stop-grpc-service-on-defrag flag to feature gate. #18359
• feature metrics added.

• Milestone 2

• e2e tests added for the feature gate equivalent of the selected experimental feature.
• robustness test scenarios added for the selected experimental feature(s).
• documentation to track all feature gates added to [etcd-io/website].

• Milestone 3

• migrate all existing --experimental server level features to feature gates without removing the old flags.
  List of features to migrate:
  • experimental-stop-grpc-service-on-defrag: migrate experimental-stop-grpc-service-on-defrag flag to feature gate. #18359
  • experimental-enable-distributed-tracing: migrate --experimental-enable-distributed-tracing flag to feature gate #19050
  • experimental-initial-corrupt-check: KEP-4578: migrate experimental-initial-corrupt-check flag to feature gate. #18478
  • experimental-compact-hash-check-enabled: migrate experimental-compact-hash-check-enabled to feature gate. #19053
  • experimental-peer-skip-client-san-verification: migrate experimental-peer-skip-client-san-verification flag to feature gate #19062
  • experimental-enable-lease-checkpoint and experimental-enable-lease-checkpoint-persist: KEP-4578: migrate experimental-enable-lease-checkpoint and experimental-enable-lease-checkpoint-persist flag to feature gate. #18479
  • experimental-memory-mlock: migrate experimental-memory-mlock flag to feature gate #19061
  • experimental-txn-mode-write-with-shared-buffer: migrate experimental-txn-mode-write-with-shared-buffer flag to feature gate #19063

Why is this needed?

To make it easier to add and use new features in etcd.
Please refer to kubernetes/enhancements#4578 for more details.

[siyuanfoundation]

while kubernetes/enhancements#4610 is still in review, I think there is no much contention regarding server level feature gate. we can get started on the implementation.
/cc @stackbaek @henrybear327

[henrybear327]

Thanks @siyuanfoundation for the heads up! Happy to contribute to this feature :) Looking forward to it!

[serathius]

Can we wait for approvals for the KEP so the implementation done by contributors don't need to be updated when the design details are changed frustrating the contributors?

[siyuanfoundation]

/cc @YoyinZyc