Kubernetes EKS Error: Unable to attach or mount volumes

Posted on February 12, 2023February 12, 2023 | by rajeshkumar

Post Views: 83

Error


Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    12m                    default-scheduler  Successfully assigned default/gitlab-postgresql-0 to ip-192-168-159-35.ap-northeast-1.compute.internal
  Warning  FailedMount  10m                    kubelet            Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-glj46 custom-init-scripts postgresql-password dshm data]: timed out waiting for the condition
  Warning  FailedMount  6m13s (x2 over 8m28s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-glj46 custom-init-scripts postgresql-password dshm]: timed out waiting for the condition
  Warning  FailedMount  98s (x2 over 3m55s)    kubelet            Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[dshm data kube-api-access-glj46 custom-init-scripts postgresql-password]: timed out waiting for the condition

$ kubectl logs deployment/ebs-csi-controller -n kube-system -c ebs-plugin

        status code: 403, request id: f4bdbecb-40d5-4eeb-bcef-d0b734a94c2a
E0212 21:04:38.366854       1 driver.go:120] GRPC error: rpc error: code = Internal desc = Could not attach volume "vol-0b10c235246e76523" to node "i-0bceabf074ee5f7c7": could not attach volume "vol-0b10c235246e76523" to node "i-0bceabf074ee5f7c7": UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 1rf720y-vwEYGFNwphni8ZXNr42fkuH3Vx7GWJgExmOd58-tN0S4nkAG6RHWPjHCl_ODo4ripUzogFRKRyPbFOROFCzl7uyTgs3RcWrVVWX0Ug6scvyKRvO7SPMhXsWH0HpDPXWJhqo1_9hJzgP13hE1ecfqCsN204zQQNYziNf3dmELgHnW24XQMdDEF_TOzY0u82xBRJUIVvb7W-w7E1PWbYCW0pT_D8AuEIeoRY-fXfmGZb11-SqY35GB1wFBt-06s0tqphQbthMuRLT5ios33FcyJE3PqI2o6FHF09CGnbFcoxCR1BaDKZ7RAIxM_qHP87JuOSZvQxk3lYa45rlqhj3p0dI4ByTVO1sNX6EJFLkffAnLa0-GSbRhWubUlj1bPQ_UqYnkK5iII2h4IBIUvrPu0vHR0tAkdb2BIM1r7vl1vx9KPFUfjXMhu_KA7thujWYwb7_9N3pj-VC4nn8SL5gmtWqB9NdUziSLh76WlA9xmuB59fJOoFVFdsvmawMxFM3rKCrmHFJUiot9-ZcrC9adZe6wPu4CVqA_Coqm_IIuPc6haySr6P_EylT4k51Bo08eUWCaSQilRFYwEh0GlN4cqOSaiEJ6hGhRg1ID_Qgxt1Iz3kM00hlRBPO3JIYzQY3k-24vvhBZShUmO8fa2MkAIhBArdSwTVnhb0kt3R-unLNkyguWJ8A
        status code: 403, request id: c6f0488d-0a45-4e70-bb99-35c3635418a6

Solution

I figured what my issue was. My AWS EBS CSI controllers were running on nodes with IAM roles having insufficient permissions.
As a result I was seeing these messages in the logs:

So I had to:

add AmazonEBSCSIDriverPolicy policy to the IAM role
adjust my HELM chart vars and include controller.nodeSelector.ops="true" option to make
it run on the nodes with that IAM role.

so my aws ebs csi driver helm chart values

# https://github.com/kubernetes-sigs/aws-ebs-csi-driver
node:
  # tolerateAllTaints: true
  tolerations:
    - effect: NoSchedule
      operator: Exists
 
controller:
  nodeSelector:
    ops: "true"

storageClasses:
  - allowVolumeExpansion: true
    allowedTopologies:
    - matchLabelExpressions:
      - key: topology.ebs.csi.aws.com/zone
        values:
        - us-west-2a
        - us-west-2b
        - us-west-2c
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
    name: gp3
    parameters:
      csi.storage.k8s.io/fstype: ext4
      type: gp3
    provisioner: ebs.csi.aws.com
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer