Enhancing NodeIPAM to support multiple ClusterCIDRs

[rahulkjoshi]

Enhancement Description

• One-line enhancement description (can be used as a release note): Enhancing NodeIPAM controller to support multiple ClusterCIDRs
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2593-multiple-cluster-cidrs
• Discussion Link: Many discussions around the future of pod cidrs:
  • Would like ClusterCIDR to be fetchable by pods. kubernetes#46508 (comment)
  • Support multiple cluster-cidrs in kube-controller-manager kubernetes#94862
• Primary contact (assignee): @sarveshr7
• Responsible SIGs: network
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.25
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • Support Multiple Cluster CIDRs #2594
    • KEP:2593:Update ClusterCIDRConfig API, Test plan and Milestones #3402
  • Code (k/k) update PR(s):
    • Enhance NodeIPAM to support multiple ClusterCIDRs kubernetes#109090
  • Docs (k/website) update PR(s):
• Beta
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

This KEP has been withdrawn #4253 , there is some remaining work on creating a new repo to host the existing code and removing it from tree

[rahulkjoshi]

/sig network

[dcbw]

[sig-network 2021-06-10] Waiting for SIG Network to decide whether to do this as a built-in or a CRD. Tim wants to see if we can define the type here as a CRD and make the controller built-in first. If we can't figure this out in the next few weeks, we'll just try to make it a built-in.

[wojtek-t]

@rahulkjoshi - to target 1.23, you need PRR approved By Thursday - feel free to assign to me once you open a PR

[salaxander]

/milestone v1.23

[salaxander]

/stage alpha

[salaxander]

Hi @rahulkjoshi! 1.23 Enhancements team here. Just checking in as we approach enhancements freeze at 11:59pm PST on Thursday 09/09. Here's where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

I see there's an open PR that will complete those last 2 items, so once that merges we're all good to go!

Thanks!

[supriya-premkumar]

Hi @rahulkjoshi, 1.23 Enhancements Shadow here 👋🏽
This issue is currently marked as atRisk and I am following up as we are approaching the enhancements freeze deadline at 23:59:59 PDT on Thursday, September 9th.

Looks like the merged KEP number doesn't match the enhancements issue #2593. It is currently pointing to the PR #. Could you please update that?

[rahulkjoshi]

@supriya-premkumar @salaxander please take a look and let me know if this PR needs any other changes.

[supriya-premkumar]

Thank you @rahulkjoshi. Looks like this enhancement is all set for the freeze party 🎉
I will update the tracking sheet.

[supriya-premkumar]

Hi @rahulkjoshi👋🏽 1.23 Enhancements Shadow here
Just checking on the PR status for this issue as we are approaching code freeze deadline on Tuesday, November 16 at 6:00 pm PST

I see that the linked PRs are merged. Are there any other associated PRs for this issue?
Also, please update the respective PRs in the description

Marking this issue as tracked. Please let me know if there are any open PRs.

[rahulkjoshi]

I think the code will not be merged before the 1.23 deadline. We'll probably merge into 1.24

@sarveshr7

[supriya-premkumar]

@rahulkjoshi Are there any open PRs? If yes, Could you please link them in the description?
Marking this issue as atRisk for now

[rahulkjoshi]

To my knowledge, there aren't any open PRs. I'll switch this to be targeting alpha for 1.24 since it seems we're missing the deadline.

[reylejano]

Thank you @rahulkjoshi for confirming that this enhancement will target 1.24
/milestone v1.24

[uablrek]

The perNodeHostBits should be per family.

To have just one for both ipv4/ipv6 enforces a limit on the number of PODs on a node only based on the shortcomings of ipv4. It should be possible to define for instance "perNodeHostBits4: 7" and "perNodeHostBits6: 32"

Some CNI-plugins, like Calico, already allows to configure ipv4 to some (not all) PODs on a node. My own kube-ipam can assign ipv4 addresses only to selected namespaces. I expect this feature, in some form, to be common practice in large cluster in the future.

It is possible to assign large segments to ipv6-only nodes in a cluster, and have a mix of dual-stack and ipv6-only nodes with the current implementation, but that is not enough.

[thockin]

For simplicity we made them the same, on the assumption that all pods have the same set of families. If that is not true, then we should think about decoupling and having per-family rules. Sigh.

…

On Mon, Jul 10, 2023 at 6:22 AM Lars Ekman ***@***.***> wrote: The perNodeHostBits should be per family. To have *just one* for both ipv4/ipv6 enforces a limit on the number of PODs on a node only based on the shortcomings of ipv4. It should be possible to define for instance "perNodeHostBits4: 7" and "perNodeHostBits6: 32" Some CNI-plugins, like Calico, already allows to configure ipv4 to *some* (not all) PODs on a node. My own kube-ipam <https://github.com/Nordix/ipam-node-annotation#limit-ipv4-address-allocation> can assign ipv4 addresses only to selected namespaces. I expect this feature, in some form, to be common practice in large cluster in the future. It is possible to assign large segments to ipv6-only nodes in a cluster, and have a mix of dual-stack and ipv6-only nodes with the current implementation, but that is not enough. — Reply to this email directly, view it on GitHub <#2593 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVA4DINY6VYNGEZMCKLXPP62LANCNFSM42F55X2A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[thockin]

Are we still shooting for Beta in 29?

[aojea]

Are we still shooting for Beta in 29?

There is concerns about moving this API inside the core group as it can cause confusion on users and the networking implementation already have their own mechanisms , see #4174

[rayandas]

Hello @rahulkjoshi 👋, v1.29 Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage beta for v1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29.
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update the latest-milestone and graduation criteria (if any).

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[BenTheElder]

I believe this should be removed from the milestone following #4253?

[thockin]

Done

…

On Fri, Sep 29, 2023 at 3:00 PM Benjamin Elder ***@***.***> wrote: I believe this should be removed from the milestone following #4253 <#4253>? — Reply to this email directly, view it on GitHub <#2593 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVFHFB56DIXNXR4ECBLX45AI3ANCNFSM42F55X2A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aojea]

Are we still shooting for Beta in 29?

There is concerns about moving this API inside the core group as it can cause confusion on users and the networking implementation already have their own mechanisms , see #4174

/close

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

Are we still shooting for Beta in 29?

There is concerns about moving this API inside the core group as it can cause confusion on users and the networking implementation already have their own mechanisms , see #4174

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

This is now implemented out of tree https://github.com/kubernetes-sigs/node-ipam-controller