[bug] network policy is not compatible with kube-proxy's config of clusterCIDR under new ippool

[luckymrwang]

Expected Behavior

under the networkpolicy : allow pod connection each other in one namespace, deny cross namespace traffic.
under one namespace bond new ippool, in one pod requesting clusterIP:port should be allowed to annother pod

Current Behavior

under the networkpolicy : allow pod connection each other in one namespace, deny cross namespace traffic.
under one namespace bond new ippool, in one pod requesting clusterIP:port is denied to annother pod

Possible Solution

1. kube-proxy config:

• Add kube-proxy config clusterCIDR containing new ippool, but the config only support one ipv4 cidr.
• Maybe change kube-proxy config clusterCIDR to 0.0.0.0/0, but it may affect request service whose endpoints is hostnetwork mode.

2. Add tunl0 IP to networkpolicy. But every node is different and the tunl0 is created dynamically.

Steps to Reproduce (for bugs)

1. create a new ippool as a subnet, such as 192.168.1.0/24 as follows

apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: subnet
spec:
  blockSize: 24
  cidr: 192.168.1.0/24
  ipipMode: Always
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never

4. create a namespace named ns1, add ns annotation "cni.projectcalico.org/ipv4pools": "192.168.1.0/24"
5. under namespace ns1:

• create a deployment named dp1 whose pod is on node1, and the pod name is like dp1-xxx, podIP is 192.168.1.1
• create annother deployment named dp2 whose pod is on node2, the pod name is like dp2-xxx and podIP is 192.168.1.2. In addition, create a service named svc2 whose selector matches pod dp2-xxx, such as svc2's clusterIP is 10.233.56.189 and port is 8080
• create a networkpolicy as follows

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: netpol1
  namespace: ns1
spec:
  egress:
  - to:
    - podSelector: {}
  ingress:
  - from:
    - podSelector: {}
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

to allow pod connection each other in one namespace, deny cross namespace traffic.

6. when kubectl -n ns1 exec -it dp1-xxx curl 10.233.56.189:8080, the traffic is denied.

Context

Reason:

1. the k8s cluster defalut ippool is 10.238.64.0/18 and the kube-proxy config of clusterCIDR is 10.238.64.0/18
2. kube-proxy create a iptable rule -A KUBE-SERVICES ! -s 10.238.64.0/18 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ in nat table
3. when kubectl -n ns1 exec -it dp1-xxx curl 10.233.56.189:8080, package will leave node1, this moment package's src IP is 192.168.1.1 not in 10.238.64.0/18 matching above iptable rule, so do SNAT chaning package's src to tunl0's IP 10.233.90.0, leaving node1.
4. to node2, calico creates iptable rule according to networkpolicy netpol1.

• such as cali-tw-calid0xxxx -m comment "cali:xxx" -m mark 0x0/0x20000 -j cali-pi-_xxxxx

• cali-pi-_xxxxx -m comment "xxxxx" -m set --match-set cali40s:xx src -j MARK --set-xmark 0x10000/0x10000

• ipset cali40s:xx contains pod dp1-xxx's IP 192.168.1.1 allowed, not contains tunl0's IP 10.233.90.0

  So the package is droped.

Your Environment

• Calico version
  v3.21.4

• Orchestrator version (e.g. kubernetes, mesos, rkt):
  kubernetes v1.20.10

• Operating System and version:
  CentOS7

• Link to your project (optional):

[mgleung]

Hi @luckymrwang , does the cluster CIDR in the kube-proxy config match your IP pool CIDRs? Reason I'm asking is that a misconfigured kube-proxy can cause NAT to occur when Calico does not expect it which would cause Calico policy to drop that traffic (more details can be found here).

[luckymrwang]

@mgleung Thanks for your reply.
The cluster CIDR in the kube-proxy config is 10.233.64.0/18. And the calico defalut ippool is also 10.233.64.0/18. The question is that I create a new ippool whose CIDR is 192.168.1.0/24 different from defalut ippool and kube-proxy config.
The two ippool are valid and used. But the kube-proxy config just only supports one ipv4 CIDR 10.233.64.0/18. As above NetworkPolicy, under CIDR 10.233.64.0/18 traffic is normal by requesting service clusterIP. But under CIDR 192.168.1.0/24 traffic is dropped by requesting service clusterIP.

[mgleung]

@luckymrwang gotcha. I think it makes sense then that the traffic in CIDR 192.168.1.0/24 is being dropped then since that's not configured for the kube-proxy so the traffic is masqueraded and then dropped by our network policy. I'm not sure if we can support traffic in disjoint CIDRs from the kube-proxy. Maybe @caseydavenport might know?

[caseydavenport]

This is unfortunately a limitation in Kubernetes - you could disable the kube-proxy clusterCIDR altogether, but then you lose out on the ability to bridge out-of-cluster traffic to the pod IP range in certain scenarios.

There is upstream work in progress to make this better, both in the form of supporting multiple cluster CIDRs as well as improving kube-proxy "local" detection to not rely on CIDR at all (the more useful fix)

• Support Multiple Cluster CIDRs kubernetes/enhancements#2594
• kube-proxy --cluster-cidr should support multiple CIDRs kubernetes/kubernetes#81675

[mgleung]

Closing this since it seems that it's a limitation of the number of cluster CIDRs that kube-proxy can be configured with. I think the correct fix should be made upstream unless anyone has any objections?