Error
kubectl describe pod istiod-7957f87bcc-k6c94 -nistio-system
Name: istiod-7957f87bcc-k6c94
Namespace: istio-system
Priority: 0
Service Account: istiod
Node: <none>
Labels: app=istiod
install.operator.istio.io/owning-resource=unknown
istio=pilot
istio.io/rev=default
operator.istio.io/component=Pilot
pod-template-hash=7957f87bcc
sidecar.istio.io/inject=false
Annotations: prometheus.io/port: 15014
prometheus.io/scrape: true
sidecar.istio.io/inject: false
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/istiod-7957f87bcc
Containers:
discovery:
Image: docker.io/istio/pilot:1.16.3
Ports: 8080/TCP, 15010/TCP, 15017/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
discovery
--monitoringAddr=:15014
--log_output_level=default:info
--domain
cluster.local
--keepaliveMaxServerConnectionAge
30m
Requests:
cpu: 500m
memory: 2Gi
Readiness: http-get http://:8080/ready delay=1s timeout=5s period=3s #success=1 #failure=3
Environment:
REVISION: default
JWT_POLICY: third-party-jwt
PILOT_CERT_PROVIDER: istiod
POD_NAME: istiod-7957f87bcc-k6c94 (v1:metadata.name)
POD_NAMESPACE: istio-system (v1:metadata.namespace)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
KUBECONFIG: /var/run/secrets/remote/config
PILOT_TRACE_SAMPLING: 1
PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_OUTBOUND: true
PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_INBOUND: true
ISTIOD_ADDR: istiod.istio-system.svc:15012
PILOT_ENABLE_ANALYSIS: false
CLUSTER_ID: Kubernetes
Mounts:
/etc/cacerts from cacerts (ro)
/var/run/secrets/istio-dns from local-certs (rw)
/var/run/secrets/istiod/ca from istio-csr-ca-configmap (ro)
/var/run/secrets/istiod/tls from istio-csr-dns-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7j7hv (ro)
/var/run/secrets/remote from istio-kubeconfig (ro)
/var/run/secrets/tokens from istio-token (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
local-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
cacerts:
Type: Secret (a volume populated by a Secret)
SecretName: cacerts
Optional: true
istio-kubeconfig:
Type: Secret (a volume populated by a Secret)
SecretName: istio-kubeconfig
Optional: true
istio-csr-dns-cert:
Type: Secret (a volume populated by a Secret)
SecretName: istiod-tls
Optional: true
istio-csr-ca-configmap:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: true
kube-api-access-7j7hv:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 12m (x4 over 27m) default-scheduler 0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Warning FailedScheduling 4m58s (x3 over 8m21s) default-scheduler 0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.Solution
Based on the output of kubectl describe pod for the istiod-7957f87bcc-k6c94 pod in the istio-system namespace, the pod is currently in a Pending state and has not been scheduled to any node due to insufficient resources on the available nodes.
The events section shows that the pod failed to schedule four times with a FailedScheduling warning. The first time it failed was due to insufficient memory, and subsequent failures were due to both insufficient CPU and memory.
The node selectors and tolerations sections show that there are no specific node selectors or tolerations defined for the pod.
To resolve this issue, you may need to increase the resources available on the existing nodes, or add new nodes to the cluster. Alternatively, you can consider lowering the resource requests of the pod or modifying the resource limits of other pods running on the same nodes to free up resources for this pod.The error message “0/1 nodes are available: 1 No preemption victims found for incoming pod” indicates that the Kubernetes scheduler is unable to schedule a new pod on any of the available nodes because there are not enough resources available to meet the pod’s requirements.
This error message usually occurs when there are not enough resources available on the nodes to meet the requirements of the pod, such as CPU, memory, or other resource requests specified in the pod’s configuration.
To resolve this issue, you can try one or more of the following steps:
- Check the resource requests and limits of the pod and make sure they are appropriate for the available resources on the nodes.
- Check the resource usage of the existing pods on the nodes and make sure there is enough free resources available for the new pod.
- Increase the resources available on the nodes by adding more nodes to the cluster or by upgrading the existing nodes.
- Enable pod preemption in the Kubernetes scheduler to allow lower priority pods to be evicted from the nodes to make room for higher priority pods.
- Adjust the priority of the pod to ensure it is scheduled before lower priority pods.
Once you have made the necessary changes, you can try to reschedule the pod by deleting and recreating it or by updating its configuration.
Latest posts by rajeshkumar (see all)