Error
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned default/gitlab-postgresql-0 to ip-192-168-159-35.ap-northeast-1.compute.internal
Warning FailedMount 10m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-glj46 custom-init-scripts postgresql-password dshm data]: timed out waiting for the condition
Warning FailedMount 6m13s (x2 over 8m28s) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-glj46 custom-init-scripts postgresql-password dshm]: timed out waiting for the condition
Warning FailedMount 98s (x2 over 3m55s) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[dshm data kube-api-access-glj46 custom-init-scripts postgresql-password]: timed out waiting for the condition$ kubectl logs deployment/ebs-csi-controller -n kube-system -c ebs-plugin
status code: 403, request id: f4bdbecb-40d5-4eeb-bcef-d0b734a94c2a
E0212 21:04:38.366854 1 driver.go:120] GRPC error: rpc error: code = Internal desc = Could not attach volume "vol-0b10c235246e76523" to node "i-0bceabf074ee5f7c7": could not attach volume "vol-0b10c235246e76523" to node "i-0bceabf074ee5f7c7": UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 1rf720y-vwEYGFNwphni8ZXNr42fkuH3Vx7GWJgExmOd58-tN0S4nkAG6RHWPjHCl_ODo4ripUzogFRKRyPbFOROFCzl7uyTgs3RcWrVVWX0Ug6scvyKRvO7SPMhXsWH0HpDPXWJhqo1_9hJzgP13hE1ecfqCsN204zQQNYziNf3dmELgHnW24XQMdDEF_TOzY0u82xBRJUIVvb7W-w7E1PWbYCW0pT_D8AuEIeoRY-fXfmGZb11-SqY35GB1wFBt-06s0tqphQbthMuRLT5ios33FcyJE3PqI2o6FHF09CGnbFcoxCR1BaDKZ7RAIxM_qHP87JuOSZvQxk3lYa45rlqhj3p0dI4ByTVO1sNX6EJFLkffAnLa0-GSbRhWubUlj1bPQ_UqYnkK5iII2h4IBIUvrPu0vHR0tAkdb2BIM1r7vl1vx9KPFUfjXMhu_KA7thujWYwb7_9N3pj-VC4nn8SL5gmtWqB9NdUziSLh76WlA9xmuB59fJOoFVFdsvmawMxFM3rKCrmHFJUiot9-ZcrC9adZe6wPu4CVqA_Coqm_IIuPc6haySr6P_EylT4k51Bo08eUWCaSQilRFYwEh0GlN4cqOSaiEJ6hGhRg1ID_Qgxt1Iz3kM00hlRBPO3JIYzQY3k-24vvhBZShUmO8fa2MkAIhBArdSwTVnhb0kt3R-unLNkyguWJ8A
status code: 403, request id: c6f0488d-0a45-4e70-bb99-35c3635418a6Solution
I figured what my issue was. My AWS EBS CSI controllers were running on nodes with IAM roles having insufficient permissions.
As a result I was seeing these messages in the logs:
So I had to:
- add AmazonEBSCSIDriverPolicy policy to the IAM role
- adjust my HELM chart vars and include
controller.nodeSelector.ops="true"option to make
it run on the nodes with that IAM role.
so my aws ebs csi driver helm chart values
# https://github.com/kubernetes-sigs/aws-ebs-csi-driver
node:
# tolerateAllTaints: true
tolerations:
- effect: NoSchedule
operator: Exists
controller:
nodeSelector:
ops: "true"
storageClasses:
- allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
- key: topology.ebs.csi.aws.com/zone
values:
- us-west-2a
- us-west-2b
- us-west-2c
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: gp3
parameters:
csi.storage.k8s.io/fstype: ext4
type: gp3
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer