[Lima? CNI? (w/ rootless?)] network degrading over time?

[apostasie]

Is there a network guru here who could advise on how to further debug this?

Description

After heavy, prolonged usage and testing of nerdctl, network inside lima seems to be degrading, with a very large proportion of all requests ending with i/o timeout.

This is affecting the entire VM networking, not just nerdctl. Rebooting the VM does not help.

The same requests ran from the host (or from another VM) are just fine.

It is unclear to me if this would be a lima issue, a cni issue, or a nerdctl issue?

Something as simple as
curl https://ghcr.io/v2/stargz-containers/registry/manifests/2-org

Will intermittently (~70% of the time) fail with:

curl: (28) Failed to connect to ghcr.io port 443 after 132561 ms: Couldn't connect to server

tcpdump:

11:56:15.724007 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029123835 ecr 0,nop,wscale 7], length 0
11:56:16.769207 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029124880 ecr 0,nop,wscale 7], length 0
11:56:17.794140 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029125905 ecr 0,nop,wscale 7], length 0
11:56:18.812797 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029126924 ecr 0,nop,wscale 7], length 0
11:56:19.842554 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029127953 ecr 0,nop,wscale 7], length 0
11:56:20.860340 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029128971 ecr 0,nop,wscale 7], length 0
11:56:22.906894 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029131018 ecr 0,nop,wscale 7], length 0
11:56:26.942212 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029135053 ecr 0,nop,wscale 7], length 0
11:56:35.133635 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029143245 ecr 0,nop,wscale 7], length 0
11:56:51.515252 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029159626 ecr 0,nop,wscale 7], length 0
11:57:23.775608 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029191886 ecr 0,nop,wscale 7], length 0

iptables-save

# Generated by iptables-save v1.8.10 (nf_tables) on Wed Oct  2 12:00:27 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
:CNI-ISOLATION-STAGE-1 - [0:0]
:CNI-ISOLATION-STAGE-2 - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j CNI-ISOLATION-STAGE-1
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin admin overrides" -j CNI-ADMIN
-A CNI-ISOLATION-STAGE-1 -i nerdctl0 ! -o nerdctl0 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j CNI-ISOLATION-STAGE-2
-A CNI-ISOLATION-STAGE-1 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j RETURN
-A CNI-ISOLATION-STAGE-2 -o nerdctl0 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j DROP
-A CNI-ISOLATION-STAGE-2 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j RETURN
COMMIT
# Completed on Wed Oct  2 12:00:27 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Wed Oct  2 12:00:27 2024
*nat
:PREROUTING ACCEPT [4:1843]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [72782:32720052]
:POSTROUTING ACCEPT [72782:32720052]
:CNI-53bc5ebfdf1a5ca6fc355b8a - [0:0]
:CNI-bca742bf74f55524d8dda11b - [0:0]
:LIMADNS - [0:0]
-A PREROUTING -j LIMADNS
-A OUTPUT -j LIMADNS
-A POSTROUTING -s 10.4.0.21/32 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j CNI-bca742bf74f55524d8dda11b
-A POSTROUTING -s 10.4.0.22/32 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j CNI-53bc5ebfdf1a5ca6fc355b8a
-A CNI-53bc5ebfdf1a5ca6fc355b8a -d 10.4.0.0/24 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j ACCEPT
-A CNI-53bc5ebfdf1a5ca6fc355b8a ! -d 224.0.0.0/4 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j MASQUERADE
-A CNI-bca742bf74f55524d8dda11b -d 10.4.0.0/24 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j ACCEPT
-A CNI-bca742bf74f55524d8dda11b ! -d 224.0.0.0/4 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j MASQUERADE
COMMIT
# Completed on Wed Oct  2 12:00:27 2024

[apostasie]

tracepath ghcr.io
 1?: [LOCALHOST]                      pmtu 1500
 1:  no reply
 2:  no reply
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply

 ip route list table all
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 100
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 100
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 100
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 192.168.5.15 dev eth0 table local proto kernel scope host src 192.168.5.15
broadcast 192.168.5.255 dev eth0 table local proto kernel scope link src 192.168.5.15
fe80::/64 dev eth0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::5055:55ff:fefe:6a03 dev eth0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium

[AkihiroSuda]

This is affecting the entire VM networking, not just nerdctl. Rebooting the VM does not help.

Sounds like a Lima issue?

[apostasie]

This is affecting the entire VM networking, not just nerdctl. Rebooting the VM does not help.

Sounds like a Lima issue?

I am now thinking this might be a side-effect of the CNI bridge iptable issue.