@ellistarn if we have multiple available subnets available for a zone, will karpenter choose the least full subnet like it says in here https://karpenter.sh/v0.32/concepts/nodeclasses/#specsubnetselectorterms ?

Correct -- this is a good point. We don't do this for subnets in different zones.

Would be really good if we could do this, as this would/could also naturally balance instances across zones which would be a nice HA feature. We have seen that unless there is a topology spread in our deployments, multiple instances get spun up in the same zone.

This is one of the burning issue that we are currently facing with Karpenter, we discussed about this in reinvent. Checking available IPs across the subnets using AWS endpoint and scheduling next instances to the subnet with most IPs seems straight forward. Any approximate release timeline for when we can expect this feature to be added?

Are you finding that there are 0 remaining IPs and the launch fails, or just a few IPs? Can you provide logs that occur when this happens? Is the failure at the node level? Pod level?

Are you finding that there are 0 remaining IPs and the launch fails, or just a few IPs?

No we were seeing that without pod anti affiinity to force them to be spread across zones we'd often end up with multiple nodes in the 1 zone and none in the other 2 zones, then all the pods trying to spin up and exhaust the subnet and we see pods get stuck in pending as the CNI can't assign them an IP. Note that this is using lots of small (cpu/mem) pods, relatively small subnets (/26) and pod security groups. We weren't seeing node launches fail.

We're not seeing this issue as we moved to larger subnets and added the anti affinity which means the nodes are spread across the zones/subnets more evenly.

Am happy to recreate this and send some logs if that would be helpful @ellistarn ?

@martinsmatthews , have you completely exhausted the ipv4 space? Is it possible to add another subnet with more IPs? Why are you so constrained?

I'm working on an idea for EKS networking and I'd love to chat more over slack.

Hi @ellistarn sorry this was just an example which would highlight the issue we're discussing, we don't have this problem any more as we just gave this nodepool 3 /25s and all is fine now.

Fwiw, we run a number of different nodepools per cluster, some with quite small numbers of pods. For example the one we were seeing this issue in runs no more than 40-60 pods at any one time so 3x /26 was enough IPs to allow this. We are not exactly resource constrained but at the same time we need to think about how much internal IP space we have an allocate it sensibly as it is a finite resource.

Definitely happy to chat on Slack - will ping you.

Coming back to the use of topologySpreadConstraints to solve this issue and stripe pods over AZs (and thus over subnets) this doesn't work perfectly. I spun up 10 deployments with 3 replicas and with a topologySpreadConstraints like (plus a large enough resource ask to end up with only one pod per node) with a nodepool that had 3 subnets defined, 1 in each of 3 AZs:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: topology-test-{{ count }}

and this gave me 30 nodes, 10 in each AZ - bingo. But... then I dropped the replica count to 2 and spun up 15 deployments and this gave me a very uneven spread of nodes:

• us-west-2a: 19
• us-west-2b: 7
• us-west-2c: 14

Obviously this is artificial, but this does again highlight a need for an option to balance nodes across AZs even if this is not the default. And not just for subnet usage reasons, there is an HA risk here for when we lose a zone - it doesn't happen often, but when it does and we were skewed like this in a production cluster, it wouldn't be pretty.

Has anyone come up with a solution to work around this? We have twice run into a situation where Karpenter overloads a single AZ (with two /20 subnets) and both subnets run out of IP's. The skew was quite extreme (200 nodes in 1 AZ, 50 in the other 3) and it seems like a burst of spot requests may have put them all on the same AZ, but it still seems odd that Karpenter doesn't attempt to do any kind of load balancing across the AZ's. Our current work around is to define a nodepool per AZ and use the cpu/memory limits to faux-limit the number of nodes to prevent IP exhaust, but this results in a lot of nodepools and ec2nodeclass's. This only works because we currently use a fairly strict set of instance types so we know what their max pod count would be. If anyone has come up with a better solution please share!

We have twice run into a situation where Karpenter overloads a single AZ (with two /20 subnets) and both subnets run out of IP's.
Same issue here.

After migrating to karpenter, it started favoring one zone (eu-west-1c), despite having over 1000 addresses available in other subnets, which ended up with us running out of addresses in eu-west-1c.

╰─❯ kubectl get nodes -L  worker-type,group,topology.kubernetes.io/zone --sort-by=.metadata.creationTimestamp --no-headers   -l group=default | awk '{print $8}'  |sort | uniq -c
   7 eu-west-1a
   7 eu-west-1b
  25 eu-west-1c

different cluster but still running on cluster-autoscaler:

18 eu-west-1a
17 eu-west-1b
18 eu-west-1c