Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pod cycle restart,Pod sandbox changed, it will be killed and re-created. #4661

Open
pruidong opened this issue Sep 4, 2024 · 5 comments
Open

pod cycle restart,Pod sandbox changed, it will be killed and re-created. #4661

pruidong opened this issue Sep 4, 2024 · 5 comments

Comments

@pruidong
Copy link

pruidong commented Sep 4, 2024 •
edited
Loading

Summary

I installed MicroK8s v1.30.4 revision 7167 on Ubuntu 24.04. Using microK8s status to check the status, MicroK8s is running, but when checking the status of pods, only the Calico node is running, and all other pods are ContainerCreating The ContainerCreating status has been going on for a long time. I have checked the detailed logs and the error reason is: Pod sandbox changed, it will be killed and re created. I used #1775 The method in the system involves stopping snap.microk8s.daemon-apiserver picker to stop the apiserver picker service, but the pod still restarts repeatedly. What should I do? thank you.

System:

Ubuntu 24.04

MicroK8s Version:

1.30.4

What Should Happen Instead?

All pods are in running state

Reproduction Steps

  1. sudo snap install microk8s --classic
  2. microk8s status --wait-ready
  3. microk8s enable dashboard
  4. microk8s enable dns
  5. microk8s kubectl get all --all-namespaces
  6. microk8s kubectl get pods --all-namespaces

Detail

> microk8s kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY   STATUS              RESTARTS      AGE
kube-system   calico-kube-controllers-796fb75cc-8h6m2      0/1     ContainerCreating   0             17h
kube-system   calico-node-cf48j                            1/1     Running             1 (16h ago)   17h
kube-system   coredns-5986966c54-65tdp                     0/1     ContainerCreating   0             17h
kube-system   dashboard-metrics-scraper-795895d745-72jrs   0/1     ContainerCreating   0             17h
kube-system   hostpath-provisioner-7c8bdf94b8-lgszw        0/1     ContainerCreating   0             17h
kube-system   kubernetes-dashboard-6796797fb5-bdrsj        0/1     ContainerCreating   0             17h
kube-system   metrics-server-7cff7889bd-hx2q6              0/1     ContainerCreating   0             17h

> microk8s version
MicroK8s v1.30.4 revision 7167

> microk8s kubectl describe pods kubernetes-dashboard-6796797fb5-bdrsj -n kube-system
Name:             kubernetes-dashboard-6796797fb5-bdrsj
Namespace:        kube-system
Priority:         0
Service Account:  kubernetes-dashboard
Node:             xxxx-pc/192.168.254.131
Start Time:       Tue, 03 Sep 2024 22:17:25 +0800
Labels:           k8s-app=kubernetes-dashboard
                  pod-template-hash=6796797fb5
Annotations:      <none>
Status:           Pending
SeccompProfile:   RuntimeDefault
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/kubernetes-dashboard-6796797fb5
Containers:
  kubernetes-dashboard:
    Container ID:  
    Image:         kubernetesui/dashboard:v2.7.0
    Image ID:      
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --auto-generate-certificates
      --namespace=kube-system
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2gs6r (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-2gs6r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age                  From     Message
  ----    ------          ----                 ----     -------
  Normal  SandboxChanged  63s (x224 over 16h)  kubelet  Pod sandbox changed, it will be killed and re-created.

> microk8s inspect

Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
FAIL:  Service snap.microk8s.daemon-apiserver-kicker is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-apiserver-kicker
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/7167/var/kubernetes/backend/localnode.yaml': No such file or directory

Building the report tarball
Report tarball is at /var/snap/microk8s/7167/inspection-report-20240904_155434.tar.gz
@ktsakalozos
Copy link
Member

Hi @pruidong could you please attach the tarball created by the inspection script so we can see the logs of the cluster? Thank you.

@akinsella-socrate
Copy link

akinsella-socrate commented Oct 30, 2024 •
edited
Loading

Hi @ktsakalozos, I get same symptoms from time to time for now almost one year on microk8s on Mac (From version 1.28 with Multipass 1.13 to 1.31 with multipass 1.14.0).

My microk8s version: MicroK8s v1.31.1 revision 7244.
The Microk8s was bootstrapped less than a week ago (On Saturday 27th).

Usually it happens after a few days. I can observe as well errors mentioning networking and calico when it happens (Because of an expired token if I am correct):

● snap.microk8s.daemon-kubelite.service - Service for snap application microk8s.daemon-kubelite
     Loaded: loaded (/etc/systemd/system/snap.microk8s.daemon-kubelite.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-10-27 23:37:04 CET; 2 days ago
   Main PID: 278011 (kubelite)
      Tasks: 21 (limit: 19071)
     Memory: 747.4M
        CPU: 2h 12min 10.103s
     CGroup: /system.slice/snap.microk8s.daemon-kubelite.service
             └─278011 /snap/microk8s/7244/kubelite --scheduler-args-file=/var/snap/microk8s/7244/args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/7244/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/7244/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/7244/args/kubelet --apiserver-args-file=/var/snap/microk8s/7244/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/7244/credentials/client.config --start-control-plane=true

Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092190  278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092252  278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"}
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092282  278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:20:49 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:49.092300  278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-cnpg-1-initdb-wn8pn" podUID="2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a"
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.102166  278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104639  278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104671  278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"}
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104695  278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:20:51 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:51.104708  278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-init-kc5tk" podUID="9d6c80d6-90bf-4f33-a809-0725de373871"
Oct 30 08:20:52 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:20:52.914986  278011 horizontal.go:275] "Unhandled Error" err="failed to compute desired number of replicas based on listed metrics for Deployment/socrate-local/socrate-saas: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API"

It seems to relate to an "invalid bearer token, service account token has expired" that I can observe in kubelite logs:

Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072446  278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966"
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072489  278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966"}
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072526  278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"0aa0609f-1413-4115-94ae-b7e1e7adc84a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:19:16 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:16.072545  278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"0aa0609f-1413-4115-94ae-b7e1e7adc84a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="kube-system/hostpath-provisioner-microk8s-vm-l6x2c" podUID="0aa0609f-1413-4115-94ae-b7e1e7adc84a"
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.071560  278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073693  278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073760  278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5"}
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073786  278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:19:20 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:20.073801  278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"26661db4649362deef93cbc02fa231bd322dd9d27155a4188aa6830ab86041a5\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-cnpg-1-initdb-wn8pn" podUID="2a22203f-ebc5-4a1b-a2ab-c58c273ddb9a"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.073937  278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076637  278011 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076661  278011 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"containerd","ID":"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982"}
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076687  278011 kuberuntime_manager.go:1079] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\""
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.076703  278011 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"9d6c80d6-90bf-4f33-a809-0725de373871\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"0a25dea8d89d25e3e86134cc0e1a3dedd99a1a519a9a9512996884ad01a06982\\\": plugin type=\\\"calico\\\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized\"" pod="socrate-local/socrate-init-kc5tk" podUID="9d6c80d6-90bf-4f33-a809-0725de373871"
Oct 30 08:19:22 microk8s-vm microk8s.daemon-kubelite[278011]: E1030 08:19:22.077419  278011 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"

When I get this issue and that I execute k describe pods ... command I get:

Events:
  Type    Reason          Age                  From     Message
  ----    ------          ----                 ----     -------
  Normal  SandboxChanged  42s (x232 over 50m)  kubelet  Pod sandbox changed, it will be killed and re-created.

daemon-container.d also has interesting logs. Looking at timestamps, you can observe 3 different periods of time where usual and then abnormal things happen (expirer bearer token):

Oct 29 17:43:06 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T17:43:06.594391846+01:00" level=info msg="RemoveContainer for \"7db2460021bb519fa6718499fe137694b60e004b1ca480a064b34810ef647651\""
Oct 29 17:43:06 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T17:43:06.602904388+01:00" level=info msg="RemoveContainer for \"7db2460021bb519fa6718499fe137694b60e004b1ca480a064b34810ef647651\" returns successfully"
Oct 29 18:49:39 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:39.682916298+01:00" level=info msg="StopContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" with timeout 30 (s)"
Oct 29 18:49:39 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:39.683873848+01:00" level=info msg="Stop container \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" with signal terminated"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.030455819+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7896dbf49-2rtwd,Uid:3f29a488-1d2e-4018-82be-da72c5f6e3a9,Namespace:kube-system,Attempt:0,}"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.149 [INFO][2554382] plugin.go 327: Calico CNI found existing endpoint: &{{WorkloadEndpoint projectcalico.org/v3} {microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0 coredns-7896dbf49- kube-system  3f29a488-1d2e-4018-82be-da72c5f6e3a9 333941 0 2024-10-29 18:49:39 +0100 CET <nil> <nil> map[k8s-app:kube-dns pod-template-hash:7896dbf49 projectcalico.org/namespace:kube-system projectcalico.org/orchestrator:k8s projectcalico.org/serviceaccount:coredns] map[] [] [] []} {k8s  microk8s-vm  coredns-7896dbf49-2rtwd eth0 coredns [] []   [kns.kube-system ksa.kube-system.coredns] cali40c65f0aba7  [{dns UDP 53 0 } {dns-tcp TCP 53 0 } {metrics TCP 9153 0 }] []}} ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.150 [INFO][2554382] k8s.go 74: Extracted identifiers for CmdAddK8s ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.187 [INFO][2554397] ipam_plugin.go 229: Calico CNI IPAM request count IPv4=1 IPv6=0 ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" HandleID="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Workload="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.193 [INFO][2554397] ipam_plugin.go 269: Auto assigning IP ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" HandleID="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Workload="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0x4000695420), Attrs:map[string]string{"namespace":"kube-system", "node":"microk8s-vm", "pod":"coredns-7896dbf49-2rtwd", "timestamp":"2024-10-29 17:49:40.187354331 +0000 UTC"}, Hostname:"microk8s-vm", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}, MaxBlocksPerHost:0, HostReservedAttrIPv4s:(*ipam.HostReservedAttr)(nil), HostReservedAttrIPv6s:(*ipam.HostReservedAttr)(nil), IntendedUse:"Workload"}
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40+01:00" level=info msg="About to acquire host-wide IPAM lock." source="ipam_plugin.go:357"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40+01:00" level=info msg="Acquired host-wide IPAM lock." source="ipam_plugin.go:372"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.194 [INFO][2554397] ipam.go 107: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'microk8s-vm'
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.196 [INFO][2554397] ipam.go 660: Looking up existing affinities for host handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.199 [INFO][2554397] ipam.go 372: Looking up existing affinities for host host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.203 [INFO][2554397] ipam.go 489: Trying affinity for 10.1.254.64/26 host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.204 [INFO][2554397] ipam.go 155: Attempting to load block cidr=10.1.254.64/26 host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.207 [INFO][2554397] ipam.go 232: Affinity is confirmed and block has been loaded cidr=10.1.254.64/26 host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.207 [INFO][2554397] ipam.go 1180: Attempting to assign 1 addresses from block block=10.1.254.64/26 handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.210 [INFO][2554397] ipam.go 1682: Creating new handle: k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.219 [INFO][2554397] ipam.go 1203: Writing block in order to claim IPs block=10.1.254.64/26 handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.230 [INFO][2554397] ipam.go 1216: Successfully claimed IPs: [10.1.254.126/26] block=10.1.254.64/26 handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.230 [INFO][2554397] ipam.go 847: Auto-assigned 1 out of 1 IPv4s: [10.1.254.126/26] handle="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" host="microk8s-vm"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40+01:00" level=info msg="Released host-wide IPAM lock." source="ipam_plugin.go:378"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.230 [INFO][2554397] ipam_plugin.go 287: Calico CNI IPAM assigned addresses IPv4=[10.1.254.126/26] IPv6=[] ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" HandleID="k8s-pod-network.ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Workload="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.232 [INFO][2554382] k8s.go 383: Populated endpoint ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0", GenerateName:"coredns-7896dbf49-", Namespace:"kube-system", SelfLink:"", UID:"3f29a488-1d2e-4018-82be-da72c5f6e3a9", ResourceVersion:"333941", Generation:0, CreationTimestamp:time.Date(2024, time.October, 29, 18, 49, 39, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-dns", "pod-template-hash":"7896dbf49", "projectcalico.org/namespace":"kube-system", "projectcalico.org/orchestrator":"k8s", "projectcalico.org/serviceaccount":"coredns"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"microk8s-vm", ContainerID:"", Pod:"coredns-7896dbf49-2rtwd", Endpoint:"eth0", ServiceAccountName:"coredns", IPNetworks:[]string{"10.1.254.126/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.kube-system", "ksa.kube-system.coredns"}, InterfaceName:"cali40c65f0aba7", MAC:"", Ports:[]v3.WorkloadEndpointPort{v3.WorkloadEndpointPort{Name:"dns", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"UDP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"dns-tcp", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"metrics", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x23c1, HostPort:0x0, HostIP:""}}, AllowSpoofedSourcePrefixes:[]string(nil)}}
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.232 [INFO][2554382] k8s.go 384: Calico CNI using IPs: [10.1.254.126/32] ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.232 [INFO][2554382] dataplane_linux.go 68: Setting the host side veth name to cali40c65f0aba7 ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.234 [INFO][2554382] dataplane_linux.go 473: Disabling IPv4 forwarding ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.278 [INFO][2554382] k8s.go 411: Added Mac, interface name, and active container ID to endpoint ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0", GenerateName:"coredns-7896dbf49-", Namespace:"kube-system", SelfLink:"", UID:"3f29a488-1d2e-4018-82be-da72c5f6e3a9", ResourceVersion:"333941", Generation:0, CreationTimestamp:time.Date(2024, time.October, 29, 18, 49, 39, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-dns", "pod-template-hash":"7896dbf49", "projectcalico.org/namespace":"kube-system", "projectcalico.org/orchestrator":"k8s", "projectcalico.org/serviceaccount":"coredns"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"microk8s-vm", ContainerID:"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe", Pod:"coredns-7896dbf49-2rtwd", Endpoint:"eth0", ServiceAccountName:"coredns", IPNetworks:[]string{"10.1.254.126/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.kube-system", "ksa.kube-system.coredns"}, InterfaceName:"cali40c65f0aba7", MAC:"56:82:52:7a:b1:60", Ports:[]v3.WorkloadEndpointPort{v3.WorkloadEndpointPort{Name:"dns", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"UDP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"dns-tcp", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x35, HostPort:0x0, HostIP:""}, v3.WorkloadEndpointPort{Name:"metrics", Protocol:numorstring.Protocol{Type:1, NumVal:0x0, StrVal:"TCP"}, Port:0x23c1, HostPort:0x0, HostIP:""}}, AllowSpoofedSourcePrefixes:[]string(nil)}}
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:40.287 [INFO][2554382] k8s.go 489: Wrote updated endpoint to datastore ContainerID="ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe" Namespace="kube-system" Pod="coredns-7896dbf49-2rtwd" WorkloadEndpoint="microk8s--vm-k8s-coredns--7896dbf49--2rtwd-eth0"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.310464553+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.310634054+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.310646221+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.313131701+01:00" level=info msg="starting signal loop" namespace=k8s.io path=/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe pid=2554432 runtime=io.containerd.runc.v2
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.349406220+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7896dbf49-2rtwd,Uid:3f29a488-1d2e-4018-82be-da72c5f6e3a9,Namespace:kube-system,Attempt:0,} returns sandbox id \"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe\""
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.351855699+01:00" level=info msg="CreateContainer within sandbox \"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe\" for container &ContainerMetadata{Name:coredns,Attempt:0,}"
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.367374541+01:00" level=info msg="CreateContainer within sandbox \"ef161d3f9896aa3c85efeaccd7ce1a636af6f5dc60b2e58b1418bf9a092081fe\" for &ContainerMetadata{Name:coredns,Attempt:0,} returns container id \"8f83898d31719f4ef8dd6b96eee44a783275b4c7f4a6052275ec87b5235a092a\""
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.367810045+01:00" level=info msg="StartContainer for \"8f83898d31719f4ef8dd6b96eee44a783275b4c7f4a6052275ec87b5235a092a\""
Oct 29 18:49:40 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:40.401905754+01:00" level=info msg="StartContainer for \"8f83898d31719f4ef8dd6b96eee44a783275b4c7f4a6052275ec87b5235a092a\" returns successfully"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.786493665+01:00" level=info msg="shim disconnected" id=7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.786535498+01:00" level=warning msg="cleaning up after shim disconnected" id=7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0 namespace=k8s.io
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.786543207+01:00" level=info msg="cleaning up dead shim"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.791813960+01:00" level=warning msg="cleanup warnings time=\"2024-10-29T18:49:44+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2554683 runtime=io.containerd.runc.v2\n"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.796758294+01:00" level=info msg="StopContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" returns successfully"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.797232798+01:00" level=info msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\""
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.797308382+01:00" level=info msg="Container to stop \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.826756468+01:00" level=info msg="shim disconnected" id=3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.826956553+01:00" level=warning msg="cleaning up after shim disconnected" id=3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d namespace=k8s.io
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.826966261+01:00" level=info msg="cleaning up dead shim"
Oct 29 18:49:44 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44.832047472+01:00" level=warning msg="cleanup warnings time=\"2024-10-29T18:49:44+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2554714 runtime=io.containerd.runc.v2\n"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.872 [INFO][2554742] k8s.go 576: Cleaning up netns ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.872 [INFO][2554742] dataplane_linux.go 524: Deleting workload's device in netns. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" iface="eth0" netns="/var/run/netns/cni-eeebfd87-37c4-6db9-dad8-b61ec4390400"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.872 [INFO][2554742] dataplane_linux.go 535: Entered netns, deleting veth. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" iface="eth0" netns="/var/run/netns/cni-eeebfd87-37c4-6db9-dad8-b61ec4390400"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.946 [INFO][2554742] dataplane_linux.go 569: Deleted device in netns. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" after=73.601047ms iface="eth0" netns="/var/run/netns/cni-eeebfd87-37c4-6db9-dad8-b61ec4390400"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.946 [INFO][2554742] k8s.go 583: Releasing IP address(es) ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.946 [INFO][2554742] utils.go 195: Calico CNI releasing IP address ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.965 [INFO][2554761] ipam_plugin.go 416: Releasing address using handleID ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" HandleID="k8s-pod-network.3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" Workload="microk8s--vm-k8s-coredns--7896dbf49--l5bpm-eth0"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44+01:00" level=info msg="About to acquire host-wide IPAM lock." source="ipam_plugin.go:357"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:44+01:00" level=info msg="Acquired host-wide IPAM lock." source="ipam_plugin.go:372"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.998 [INFO][2554761] ipam_plugin.go 435: Released address using handleID ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" HandleID="k8s-pod-network.3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" Workload="microk8s--vm-k8s-coredns--7896dbf49--l5bpm-eth0"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:44.998 [INFO][2554761] ipam_plugin.go 444: Releasing address using workloadID ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" HandleID="k8s-pod-network.3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d" Workload="microk8s--vm-k8s-coredns--7896dbf49--l5bpm-eth0"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45+01:00" level=info msg="Released host-wide IPAM lock." source="ipam_plugin.go:378"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: 2024-10-29 18:49:45.001 [INFO][2554742] k8s.go 589: Teardown processing complete. ContainerID="3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.002903477+01:00" level=info msg="TearDown network for sandbox \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\" successfully"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.002927686+01:00" level=info msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\" returns successfully"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.748839160+01:00" level=info msg="RemoveContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\""
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.754594751+01:00" level=info msg="RemoveContainer for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" returns successfully"
Oct 29 18:49:45 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-29T18:49:45.755135381+01:00" level=error msg="ContainerStatus for \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"7c461589510f94afc569cfac5837f2b4ecba2a49e321282a36f68e7ec60ca2f0\": not found"
Oct 30 07:40:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:52.972005685+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-mq89l,Uid:a5ca97f8-e262-44c7-a5e1-677a73531613,Namespace:kube-system,Attempt:0,}"
Oct 30 07:40:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:52.978256196+01:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-l6x2c,Uid:0aa0609f-1413-4115-94ae-b7e1e7adc84a,Namespace:kube-system,Attempt:0,}"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.061891021+01:00" level=error msg="Failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\"" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.062489110+01:00" level=error msg="Failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\"" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063047364+01:00" level=error msg="encountered an error cleaning up failed sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063107573+01:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-mq89l,Uid:a5ca97f8-e262-44c7-a5e1-677a73531613,Namespace:kube-system,Attempt:0,} failed, error" error="failed to setup network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063361450+01:00" level=error msg="encountered an error cleaning up failed sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\", marking sandbox state as SANDBOX_UNKNOWN" error="plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.063388742+01:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:hostpath-provisioner-microk8s-vm-l6x2c,Uid:0aa0609f-1413-4115-94ae-b7e1e7adc84a,Namespace:kube-system,Attempt:0,} failed, error" error="failed to setup network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.301356443+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.301518653+01:00" level=info msg="Ensure that sandbox 1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966 in task-service has been cleanup successfully"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.302520536+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.302665579+01:00" level=info msg="Ensure that sandbox 6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35 in task-service has been cleanup successfully"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.333379253+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:40:53 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:40:53.336035984+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:04 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:04.045217580+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:04 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:04.084340824+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:07 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:07.796526994+01:00" level=info msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\""
Oct 30 07:41:07 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:07.826715372+01:00" level=error msg="StopPodSandbox for \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\" failed" error="failed to destroy network for sandbox \"3dd715513844ca0dabcd838a23a171b334aca783d5d91f9089cfa65372b6096d\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:08 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:08.048635688+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:08 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:08.076329212+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:17 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:17.042324007+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:17 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:17.082576210+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:22 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:22.050095281+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:22 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:22.078859166+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:29 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:29.043485545+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:29 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:29.073175607+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:35 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:35.045591333+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:35 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:35.077397566+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:42 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:42.044458439+01:00" level=info msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\""
Oct 30 07:41:42 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:42.071405788+01:00" level=error msg="StopPodSandbox for \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\" failed" error="failed to destroy network for sandbox \"6c036dbabde6cdcc8d3f6438056faea00593ee5f68f914c3487b07ccba3f0c35\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:48 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:48.043846844+01:00" level=info msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\""
Oct 30 07:41:48 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:48.098057521+01:00" level=error msg="StopPodSandbox for \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\" failed" error="failed to destroy network for sandbox \"1a096f789a8dd1588ee0990fe6c633550dd18a63b83b083e25b321b6f52a5966\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.909196319+01:00" level=info msg="StopContainer for \"93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e\" with timeout 30 (s)"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.911117520+01:00" level=info msg="Stop container \"93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e\" with signal terminated"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.977787809+01:00" level=info msg="shim disconnected" id=93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.977854435+01:00" level=warning msg="cleaning up after shim disconnected" id=93e10afd2c3846cb1e952f1136283ff8200d7b8a4afbacb0591348b83b870d5e namespace=k8s.io
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.977869602+01:00" level=info msg="cleaning up dead shim"
Oct 30 07:41:52 microk8s-vm microk8s.daemon-containerd[4255]: time="2024-10-30T07:41:52.984172008+01:00" level=warning msg="cleanup warnings time=\"2024-10-30T07:41:52+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2558950 runtime=io.containerd.runc.v2\n"

Does it help ?

@akinsella-socrate
Copy link

One additional info: I finally decided to reboot the VM (multipass shell, then sudo reboot / No worry, it is a dev env), and it seems to fix my problem, I can start pods again after the reboot. I assume that start phase of k8s parts renew the bearer token that causes problems.
Still, I don't understand to which token it relates. If anyone can give me clarity would be awesome.

@qdeabreu
Copy link

I have the same problem, @pruidong have you find a solution?

@ktsakalozos here are my inspection report:
inspection-report-20241115_150550.tar.gz

Reboot the machine does nothing with this matter for me.

@qdeabreu
Copy link

qdeabreu commented Nov 15, 2024 •
edited
Loading

Ok, found this https://stackoverflow.com/questions/66516049/microk8s-pods-are-restarting-frequently-on-my-raspberry-pi-ubuntu
Disabling ipv6 worked for me. Hope it helps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ktsakalozos @pruidong @akinsella-socrate @qdeabreu