Introduce compatibility version for Kubernetes control-plane upgrades

[logicalhan]

Enhancement Description

• One-line enhancement description (can be used as a release note):

Introduce compatibility version in Kubernetes components to enhance Kubernetes control-plane upgrades. See safer Kubernetes upgrades for more details.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/4330-compatibility-versions

• Primary contact (assignee): @logicalhan, @jpbetz, @liggitt

• Responsible SIGs: SIG api-machinery

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.31
  • Beta release target (x.y):
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s): KEP-4330: Compatibility versions #4395
  • Code (k/k) update PR(s):
    • apimachinery: API Emulation Versioning kubernetes#122891
    • [WIP] Storage version compatibilty mode kubernetes#122301
    • add emulated-version flag to kube-scheduler to control the feature gate. kubernetes#125769
    • add emulated-version flag to kube-controller-manager to control the feature gate. kubernetes#125778
    • [sample-apiserver] Fix: Use Correct Effective Version for kube kubernetes#125941
  • Docs (k/website) update PR(s):
    • Not needed. See Add doc for CompatibilityVersion. website#47007

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[jeremyrickard]

/sig api-machinery

[jpbetz]

/sig architecture

[jpbetz]

/milestone 1.30

[k8s-ci-robot]

@jpbetz: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone 1.30

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jpbetz]

/milestone v1.30

[johnbelamaric]

/lead-opted-in

[jpbetz]

/label lead-opted-in

[mickeyboxell]

Hello @logicalhan 👋, Enhancements team here.

Just checking in as we approach enhancements freeze today ([02:00 UTC Friday 9th February 2024 / 18:00 PDT Thursday 8th February 2024](https://everytimezone.com/s/1ade3dca)):.

This enhancement is targeting for stage alpha for v1.30 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the [latest template](https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template) has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.30. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check [here](https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md#submitting-a-kep-for-production-readiness-approval)).

• Make sure to update your status to implementable in the kep.yaml file.
• Please let me know when this enhancement is

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[mickeyboxell]

Hello 👋, v1.30 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in {current release}, please file an exception request. Thanks!

• It looks like you still need to update the KEP status to implementable for latest-milestone: 1.30.

[salehsedghpour]

/milestone clear

[jpbetz]

This is due to a clerical error. #4395 was intended to be the KEP PR but we neglected to notice is was still marked as provisional. #4502 fixes this.

[salehsedghpour]

@jpbetz Please make sure that #4502 is being merged and also consider filing an exception request.

[jpbetz]

@jpbetz Please make sure that #4502 is being merged and also consider filing an exception request.

#4502 is merged. Do we need a full exception for a clerical error like this?

[sreeram-venkitesh]

@siyuanfoundation Hi again! We're one week away from code freeze and I wanted to ping you here a slight reminder. Are there any other code PRs that needs to be merged for this KEP other than the ones listed below?

• apimachinery: API Emulation Versioning kubernetes#122891
• add emulated-version flag to kube-scheduler to control the feature gate. kubernetes#125769

From what I saw, looks like kubernetes/kubernetes#126037 also needs to be merged since its linked to kubernetes/kubernetes#125980.

Please let me know if there are any other PRs which we should be tracking! Thanks!

[siyuanfoundation]

kubernetes/kubernetes#126037

Hi @sreeram-venkitesh , thanks for checking in.
kubernetes/kubernetes#126037 is not a blocker in 1.31, kubernetes/kubernetes#125980 is opened to track tasks for the next release.

However, kubernetes/kubernetes#125941 needs to be merged before the code freeze.

[sreeram-venkitesh]

@siyuanfoundation Thank you for the quick response! I have edited the above message to add kubernetes/kubernetes#125941 to the PRs that needs to be tracked. Please update the issue description also when you get time! Thanks a lot again!

[siyuanfoundation]

@hailkomputer
Since Compatibility Version will be no-op in 1.31, it is still a little premature to write a blogpost about it. I have closed the blogpost placeholder PR for now. Will resume in 1.32.

[sreeram-venkitesh]

Hello @siyuanfoundation 👋 v1.31 Enhancements team here,

Unfortunately, the implementation (code related) PR(s) associated with this enhancement is not in the merge-ready state by code-freeze and hence this enhancement is now removed from the v1.31 milestone.

If you still wish to progress this enhancement in v1.31, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

/milestone clear

[jpbetz]

@sreeram-venkitesh With the exception granted and the code all merged, may we add the milestone back on to this issue?

[sreeram-venkitesh]

/milestone v1.31

[sreeram-venkitesh]

@jpbetz I've added the milestone back and updated the status to Tracked for code freeze!

[tjons]

Hi, enhancements lead here - I inadvertently added this to the 1.32 tracking board 😀. Please readd it if you wish to progress this enhancement in 1.32.

/remove-label lead-opted-in

[sftim]

@logicalhan I think we will need docs for this eventually (although we were able to skip those for the alpha)

[zjs]

I noticed information in the KEP that seems conflicting.

From the goals:

• In beta, we intend to extend support to:
  • --emulation-version range of binaryMinorVersion..binaryMinorVersion-1
  • --min-compatibility-version to binaryMinorVersion..binaryMinorVersion-1

From the beta graduation criteria:

• Emulation version support for N-3 minor versions
• Compatibility version support for N-3 minor versions

It seems like both lines came from be46376. Based on the rest of the doc (e.g., the the example in the Feature Gate Lifecycles section) and the conversation in #4395, I believe N-3 is the intended value and that the goals section should be updated. PR to do that: #4989