Kubernestes EKS Error: Readiness probe failed /app/grpc-health-probe -addr=:50051

Post Views: 17

Error

$ kubectl describe pod aws-node-8f2tp -n=kube-system

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Pulled     2m59s                kubelet            Container image "602401143452.dkr.ecr.ap-southeast-1.amazonaws.com/amazon-k8s-cni-init:v1.12.2-eksbuild.1" already present on machine
  Normal   Created    2m59s                kubelet            Created container aws-vpc-cni-init
  Normal   Started    2m59s                kubelet            Started container aws-vpc-cni-init
  Normal   Scheduled  2m59s                default-scheduler  Successfully assigned kube-system/aws-node-8f2tp to ip-10-33-28-136.ap-southeast-1.compute.internal
  Normal   Started    2m58s                kubelet            Started container aws-node
  Warning  Unhealthy  2m52s                kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:24:51.160Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  2m47s                kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:24:56.225Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  2m42s                kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:25:01.297Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  2m34s                kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:25:09.884Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  2m24s                kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:25:19.886Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  2m14s                kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:25:29.873Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  2m4s                 kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:25:39.880Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  114s                 kubelet            Readiness probe failed: {"level":"info","ts":"2023-02-28T03:25:49.876Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Warning  Unhealthy  104s                 kubelet            Liveness probe failed: {"level":"info","ts":"2023-02-28T03:25:59.877Z","caller":"/root/sdk/go1.19.2/src/runtime/proc.go:250","msg":"timeout: failed to connect service \":50051\" within 5s"}
  Normal   Created    84s (x2 over 2m58s)  kubelet            Created container aws-node
  Normal   Pulled     84s (x2 over 2m58s)  kubelet            Container image "602401143452.dkr.ecr.ap-southeast-1.amazonaws.com/amazon-k8s-cni:v1.12.2-eksbuild.1" already present on machine
  Warning  Unhealthy  84s (x7 over 104s)   kubelet            (combined from similar events): Readiness probe errored: rpc error: code = NotFound desc = failed to exec in container: failed to load task: no running task found: task d5365d25417ee26de951d45fa2e1bb04007b61b98c3bbb54a149334131568fdc not found: not found
  Normal   Killing    84s                  kubelet            Container aws-node failed liveness probe, will be restarted

    Ready:          False
    Restart Count:  1
    Requests:
      cpu:      25m
    Liveness:   exec [/app/grpc-health-probe -addr=:50051 -connect-timeout=5s -rpc-timeout=5s] delay=60s timeout=10s period=10s #success=1 #failure=3
    Readiness:  exec [/app/grpc-health-probe -addr=:50051 -connect-timeout=5s -rpc-timeout=5s] delay=1s timeout=10s period=10s #success=1 #failure=3

Solution

C:\Users\tdg.rajesh>kubectl exec -it aws-node-8f2tp -n kube-system -- /bin/bash
Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
error: unable to upgrade connection: container not found ("aws-node")

C:\Users\tdg.rajesh>kubectl exec -it aws-node-bdd52 -n kube-system -- /bin/bash cat /host/var/log/aws-routed-eni/ipamd.log
Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
error: unable to upgrade connection: container not found ("aws-node")

Author
Recent Posts

rajeshkumar