Knative Troubleshooting Reference


$ kubectl get ksvc
NAME          URL                                            LATESTCREATED       LATESTREADY         READY   REASON
hello         http://hello.default.svc.cluster.local         hello-00001                             False   RevisionMissing
serving-app   http://serving-app.default.svc.cluster.local   serving-app-00001   serving-app-00001   False   RevisionMissing
EUR+rajekum@SAG-JZQC0F3 MINGW64 ~
$ kubectl get routes
NAME          URL                                            READY   REASON
hello         http://hello.default.svc.cluster.local         False   RevisionMissing
serving-app   http://serving-app.default.svc.cluster.local   False   RevisionMissing
$ kubectl describe ksvc hello
Status:
Conditions:
Last Transition Time:        2022-12-21T06:51:16Z
Message:                     Revision "hello-00001" failed with message: Back-off pulling image "gcr.io/knative-samples/helloworld-go@sha256:5ea96ba4b872685ff4ddb5cd8d1a97ec18c18fae79ee8df0d29f446c5efe5f50".
Reason:                      RevisionFailed
Status:                      False
Type:                        ConfigurationsReady
Last Transition Time:        2022-12-21T06:51:16Z
Message:                     Configuration "hello" does not have any ready Revision.
Reason:                      RevisionMissing
Status:                      False
Type:                        Ready
Last Transition Time:        2022-12-21T06:51:16Z
Message:                     Configuration "hello" does not have any ready Revision.
Reason:                      RevisionMissing
Status:                      False
Type:                        RoutesReady
Latest Created Revision Name:  hello-00001
Observed Generation:           1
URL:                           http://hello.default.svc.cluster.local
Events:
Type    Reason   Age   From                Message
----    ------   ----  ----                -------
Normal  Created  40m   service-controller  Created Configuration "hello"
Normal  Created  40m   service-controller  Created Route "hello"
$ kubectl describe revision
Events:
Type     Reason         Age                From                 Message
----     ------         ----               ----                 -------
Warning  InternalError  27m (x2 over 38m)  revision-controller  failed to update deployment "hello-00001-deployment": Operation cannot be fulfilled on deployments.apps "hello-00001-deployment": the object has been modified; please apply your changes to the latest version and try again
$ kubectl get deploy
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
hello-00001-deployment         0/0     0            0           40m
serving-app-00001-deployment   0/0     0            0           99m
$ kubectl describe deploy hello-00001-deployment
Available      True    MinimumReplicasAvailable
Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   hello-00001-deployment-77c8db45b6 (0/0 replicas created)
Events:
Type    Reason             Age   From                   Message
----    ------             ----  ----                   -------
Normal  ScalingReplicaSet  41m   deployment-controller  Scaled up replica set hello-00001-deployment-77c8db45b6 to 1
Normal  ScalingReplicaSet  30m   deployment-controller  Scaled down replica set hello-00001-deployment-77c8db45b6 to 0 from 1
$ kubectl get deploy
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
hello-00001-deployment         0/0     0            0           43m
serving-app-00001-deployment   0/0     0            0           102m
$ kubectl get images
NAME                                     IMAGE
hello-00001-cache-user-container         gcr.io/knative-samples/helloworld-go@sha256:5ea96ba4b872685ff4ddb5cd8d1a97ec18c18fae79ee8df0d29f446c5efe5f50
serving-app-00001-cache-user-container   index.docker.io/bsucaciu/knative-serving@sha256:42e21f8c91dda289abb271791eb85e919b504e5702d824c83bedd74006905fc3
$ kubectl get ds
No resources found in default namespace.
$ kubectl logs deployment/controller -n knative-serving | grep "error"
{"severity":"ERROR","timestamp":"2022-12-21T06:51:47.621647033Z","logger":"controller","caller":"controller/controller.go:566","message":"Reconcile error","commit":"d7e9873","knative.dev/pod":"controller-6f9fb85fbd-55bjc","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"e4667e9d-fdbf-43eb-9cfb-ec52e43a39c9","knative.dev/key":"default/hello-00001","duration":"1.204800462s","error":"failed to update deployment \"hello-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"hello-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:491"}
# Do you see any events from your KService or Deployment?
$ kubectl logs deployment/controller -n knative-serving | grep "error" | less
{"severity":"ERROR","timestamp":"2022-12-21T06:06:48.890339554Z","logger":"controller","caller":"revision/reconciler.go:302","message":"Returned an error","commit":"d7e9873","knative.dev/pod":"controller-6f9fb85fbd-55bjc","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"4fcbdbe9-78be-47f1-95fc-818a378d284d","knative.dev/key":"default/serving-app-00001","targetMethod":"ReconcileKind","error":"failed to update deployment \"serving-app-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"serving-app-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go:302\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2022-12-21T06:06:48.892347992Z","logger":"controller","caller":"controller/controller.go:566","message":"Reconcile error","commit":"d7e9873","knative.dev/pod":"controller-6f9fb85fbd-55bjc","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"4fcbdbe9-78be-47f1-95fc-818a378d284d","knative.dev/key":"default/serving-app-00001","duration":"1.699373163s","error":"failed to update deployment \"serving-app-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"serving-app-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2022-12-21T06:41:16.015403555Z","logger":"controller","caller":"revision/reconciler.go:302","message":"Returned an error","commit":"d7e9873","knative.dev/pod":"controller-6f9fb85fbd-55bjc","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"1ac948f0-f996-4335-be47-1a054f4dfe59","knative.dev/key":"default/hello-00001","targetMethod":"ReconcileKind","error":"failed to update deployment \"hello-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"hello-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go:302\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20221123011842-b78020c16606/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2022-12-21T06:41:16.018854542Z","logger":"controller","caller":"controller/controller.go:566","message":"Reconcile error","commit":"d7e9873","knative.dev/pod":"controller-6f9fb85fbd-55bjc","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"1ac948f0-f996-4335-be47-1a054f4dfe59","knative.dev/key":"default/hello-00001","duration":"648.079413ms","error":"failed to update deployment \"hello-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"hello-00001-deployment\": the ob
# What does your deployment look like?
$ kubectl get deployment -l "serving.knative.dev/service=knative-service" --output yaml
# Do you see any events from your KService or Deployment? 
$ kubectl get ksvc  
$ kubectl describe ksvc knative-service 
$ kubectl describe ksvc hello
$ kubectl describe deployment -l "serving.knative.dev/service=knative-service"
$ kubectl describe deployment -l "serving.knative.dev/service=hello"
$ kubectl get deploy
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
hello-00001-deployment         0/1     1            0           63m
serving-app-00001-deployment   0/0     0            0           121m
$ kubectl get rs
NAME                                      DESIRED   CURRENT   READY   AGE
hello-00001-deployment-77c8db45b6         1         1         0       63m
serving-app-00001-deployment-589886b858   0         0         0       122m
$ kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
hello-00001-deployment-77c8db45b6-2vjj7   0/2     ContainerCreating   0          61s
$ kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
hello-00001-deployment-77c8db45b6-2vjj7   0/2     ContainerCreating   0          78s
$ kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
hello-00001-deployment-77c8db45b6-2vjj7   0/2     ContainerCreating   0          91s
Events:
Type     Reason     Age                  From               Message
----     ------     ----                 ----               -------
Normal   Scheduled  4m32s                default-scheduler  Successfully assigned default/hello1-00001-deployment-6db779fb45-tchhr to minikube
Warning  Failed     2m4s                 kubelet            Failed to pull image "index.docker.io/scmgalaxy/nginx-devopsschoolv1@sha256:0544f1e7eb70ee9967819f1655160f5457d47aa24910a32f357c1a41aab3d20b": rpc error: code = Unknown desc = context deadline exceeded
Warning  Failed     2m4s                 kubelet            Error: ErrImagePull
Normal   Pulled     2m4s                 kubelet            Container image "gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:b87bce5143a9f7d890fdc14f00a6e50bbfaed12955f98b308bf50e33381809ec" already present on machine
Normal   Created    2m                   kubelet            Created container queue-proxy
Normal   Started    112s                 kubelet            Started container queue-proxy
Normal   Pulling    104s (x2 over 4m3s)  kubelet            Pulling image "index.docker.io/scmgalaxy/nginx-devopsschoolv1@sha256:0544f1e7eb70ee9967819f1655160f5457d47aa24910a32f357c1a41aab3d20b"
Warning  Unhealthy  31s (x9 over 104s)   kubelet            Readiness probe failed: Get "http://172.17.0.16:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  11s (x2 over 22s)    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

istio error: 1 No preemption victims found for incoming pod

Error:

$ kubectl -n istio-system get deploy
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
istio-ingressgateway   2/3     3            2           15d
istiod                 3/3     3            3           15d
$ kubectl -n istio-system get pods
NAME                                    READY   STATUS    RESTARTS      AGE
istio-ingressgateway-7f87959cf9-j4lf9   0/1     Pending   0             15d
istio-ingressgateway-7f87959cf9-smxvk   1/1     Running   7 (14m ago)   15d
istio-ingressgateway-7f87959cf9-vggz8   1/1     Running   6 (14m ago)   15d
istiod-79597b775c-ffkn2                 1/1     Running   7 (14m ago)   15d
istiod-79597b775c-wktks                 1/1     Running   0             15d
istiod-79597b775c-wrs9w                 1/1     Running   8 (14m ago)   15d
$  kubectl describe pod istio-ingressgateway-7f87959cf9-j4lf9 -n=istio-system
Warning  FailedScheduling  28m (x43 over 3h31m)   default-scheduler  0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Warning  FailedScheduling  15m (x2 over 20m)      default-scheduler  0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Warning  FailedScheduling  4m49s (x2 over 9m50s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.