{"id":9950,"date":"2023-02-12T20:47:12","date_gmt":"2023-02-12T20:47:12","guid":{"rendered":"https:\/\/www.scmgalaxy.com\/tutorials\/?p=9950"},"modified":"2023-02-12T21:10:17","modified_gmt":"2023-02-12T21:10:17","slug":"kubernetes-eks-error-unable-to-attach-or-mount-volumes","status":"publish","type":"post","link":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/","title":{"rendered":"Kubernetes EKS Error: Unable to attach or mount volumes"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Error<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>\nEvents:\n  Type     Reason       Age                    From               Message\n  ----     ------       ----                   ----               -------\n  Normal   Scheduled    12m                    default-scheduler  Successfully assigned default\/gitlab-postgresql-0 to ip-192-168-159-35.ap-northeast-1.compute.internal\n  Warning  FailedMount  10m                    kubelet            Unable to attach or mount volumes: unmounted volumes=&#91;data], unattached volumes=&#91;kube-api-access-glj46 custom-init-scripts postgresql-password dshm data]: timed out waiting for the condition\n  Warning  FailedMount  6m13s (x2 over 8m28s)  kubelet            Unable to attach or mount volumes: unmounted volumes=&#91;data], unattached volumes=&#91;data kube-api-access-glj46 custom-init-scripts postgresql-password dshm]: timed out waiting for the condition\n  Warning  FailedMount  98s (x2 over 3m55s)    kubelet            Unable to attach or mount volumes: unmounted volumes=&#91;data], unattached volumes=&#91;dshm data kube-api-access-glj46 custom-init-scripts postgresql-password]: timed out waiting for the condition<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>$ kubectl logs deployment\/ebs-csi-controller -n kube-system -c ebs-plugin\r\n\r\n        status code: 403, request id: f4bdbecb-40d5-4eeb-bcef-d0b734a94c2a\r\nE0212 21:04:38.366854       1 driver.go:120] GRPC error: rpc error: code = Internal desc = Could not attach volume \"vol-0b10c235246e76523\" to node \"i-0bceabf074ee5f7c7\": could not attach volume \"vol-0b10c235246e76523\" to node \"i-0bceabf074ee5f7c7\": UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 1rf720y-vwEYGFNwphni8ZXNr42fkuH3Vx7GWJgExmOd58-tN0S4nkAG6RHWPjHCl_ODo4ripUzogFRKRyPbFOROFCzl7uyTgs3RcWrVVWX0Ug6scvyKRvO7SPMhXsWH0HpDPXWJhqo1_9hJzgP13hE1ecfqCsN204zQQNYziNf3dmELgHnW24XQMdDEF_TOzY0u82xBRJUIVvb7W-w7E1PWbYCW0pT_D8AuEIeoRY-fXfmGZb11-SqY35GB1wFBt-06s0tqphQbthMuRLT5ios33FcyJE3PqI2o6FHF09CGnbFcoxCR1BaDKZ7RAIxM_qHP87JuOSZvQxk3lYa45rlqhj3p0dI4ByTVO1sNX6EJFLkffAnLa0-GSbRhWubUlj1bPQ_UqYnkK5iII2h4IBIUvrPu0vHR0tAkdb2BIM1r7vl1vx9KPFUfjXMhu_KA7thujWYwb7_9N3pj-VC4nn8SL5gmtWqB9NdUziSLh76WlA9xmuB59fJOoFVFdsvmawMxFM3rKCrmHFJUiot9-ZcrC9adZe6wPu4CVqA_Coqm_IIuPc6haySr6P_EylT4k51Bo08eUWCaSQilRFYwEh0GlN4cqOSaiEJ6hGhRg1ID_Qgxt1Iz3kM00hlRBPO3JIYzQY3k-24vvhBZShUmO8fa2MkAIhBArdSwTVnhb0kt3R-unLNkyguWJ8A\r\n        status code: 403, request id: c6f0488d-0a45-4e70-bb99-35c3635418a6<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Solution<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I figured what my issue was. My AWS EBS CSI controllers were running on nodes with IAM roles having insufficient permissions.<br>As a result I was seeing these messages in the logs:<br><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">So I had to:<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>add AmazonEBSCSIDriverPolicy policy to the IAM role<\/li>\n\n\n\n<li>adjust my HELM chart vars and include&nbsp;<code>controller.nodeSelector.ops=\"true\"<\/code>&nbsp;option to make<br>it run on the nodes with that IAM role.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>so my aws ebs csi driver helm chart values<\/strong><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code># https:\/\/github.com\/kubernetes-sigs\/aws-ebs-csi-driver\r\nnode:\r\n  # tolerateAllTaints: true\r\n  tolerations:\r\n    - effect: NoSchedule\r\n      operator: Exists\r\n \r\ncontroller:\r\n  nodeSelector:\r\n    ops: \"true\"\r\n\r\nstorageClasses:\r\n  - allowVolumeExpansion: true\r\n    allowedTopologies:\r\n    - matchLabelExpressions:\r\n      - key: topology.ebs.csi.aws.com\/zone\r\n        values:\r\n        - us-west-2a\r\n        - us-west-2b\r\n        - us-west-2c\r\n    annotations:\r\n      storageclass.kubernetes.io\/is-default-class: \"true\"\r\n    name: gp3\r\n    parameters:\r\n      csi.storage.k8s.io\/fstype: ext4\r\n      type: gp3\r\n    provisioner: ebs.csi.aws.com\r\n    reclaimPolicy: Delete\r\n    volumeBindingMode: WaitForFirstConsumer<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Reference<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/aws.amazon.com\/premiumsupport\/knowledge-center\/eks-troubleshoot-ebs-volume-mounts\/\">How do I troubleshoot issues with my EBS volume mounts in Amazon EKS?<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Error Solution I figured what my issue was. My AWS EBS CSI controllers were running on nodes with IAM roles [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[1],"tags":[],"class_list":["post-9950","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Kubernetes EKS Error: Unable to attach or mount volumes - scmGalaxy<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kubernetes EKS Error: Unable to attach or mount volumes - scmGalaxy\" \/>\n<meta property=\"og:description\" content=\"Error Solution I figured what my issue was. My AWS EBS CSI controllers were running on nodes with IAM roles [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/\" \/>\n<meta property=\"og:site_name\" content=\"scmGalaxy\" \/>\n<meta property=\"article:published_time\" content=\"2023-02-12T20:47:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-12T21:10:17+00:00\" \/>\n<meta name=\"author\" content=\"rajeshkumar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"rajeshkumar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/\"},\"author\":{\"name\":\"rajeshkumar\",\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/#\\\/schema\\\/person\\\/a40e0dd38feee7c2779460a804d49361\"},\"headline\":\"Kubernetes EKS Error: Unable to attach or mount volumes\",\"datePublished\":\"2023-02-12T20:47:12+00:00\",\"dateModified\":\"2023-02-12T21:10:17+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/\"},\"wordCount\":97,\"commentCount\":0,\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/\",\"url\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/\",\"name\":\"Kubernetes EKS Error: Unable to attach or mount volumes - scmGalaxy\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/#website\"},\"datePublished\":\"2023-02-12T20:47:12+00:00\",\"dateModified\":\"2023-02-12T21:10:17+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/#\\\/schema\\\/person\\\/a40e0dd38feee7c2779460a804d49361\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kubernetes EKS Error: Unable to attach or mount volumes\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/#website\",\"url\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/\",\"name\":\"scmGalaxy\",\"description\":\"A Platform for Software Engineers\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/#\\\/schema\\\/person\\\/a40e0dd38feee7c2779460a804d49361\",\"name\":\"rajeshkumar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/82337e1fc8642c7f36795aa0775af055a4f38882994004b593efd31758d4434e?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/82337e1fc8642c7f36795aa0775af055a4f38882994004b593efd31758d4434e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/82337e1fc8642c7f36795aa0775af055a4f38882994004b593efd31758d4434e?s=96&d=mm&r=g\",\"caption\":\"rajeshkumar\"},\"url\":\"https:\\\/\\\/www.scmgalaxy.com\\\/tutorials\\\/author\\\/rajeshkumar\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kubernetes EKS Error: Unable to attach or mount volumes - scmGalaxy","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/","og_locale":"en_US","og_type":"article","og_title":"Kubernetes EKS Error: Unable to attach or mount volumes - scmGalaxy","og_description":"Error Solution I figured what my issue was. My AWS EBS CSI controllers were running on nodes with IAM roles [&hellip;]","og_url":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/","og_site_name":"scmGalaxy","article_published_time":"2023-02-12T20:47:12+00:00","article_modified_time":"2023-02-12T21:10:17+00:00","author":"rajeshkumar","twitter_card":"summary_large_image","twitter_misc":{"Written by":"rajeshkumar","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/#article","isPartOf":{"@id":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/"},"author":{"name":"rajeshkumar","@id":"https:\/\/www.scmgalaxy.com\/tutorials\/#\/schema\/person\/a40e0dd38feee7c2779460a804d49361"},"headline":"Kubernetes EKS Error: Unable to attach or mount volumes","datePublished":"2023-02-12T20:47:12+00:00","dateModified":"2023-02-12T21:10:17+00:00","mainEntityOfPage":{"@id":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/"},"wordCount":97,"commentCount":0,"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/","url":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/","name":"Kubernetes EKS Error: Unable to attach or mount volumes - scmGalaxy","isPartOf":{"@id":"https:\/\/www.scmgalaxy.com\/tutorials\/#website"},"datePublished":"2023-02-12T20:47:12+00:00","dateModified":"2023-02-12T21:10:17+00:00","author":{"@id":"https:\/\/www.scmgalaxy.com\/tutorials\/#\/schema\/person\/a40e0dd38feee7c2779460a804d49361"},"breadcrumb":{"@id":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.scmgalaxy.com\/tutorials\/kubernetes-eks-error-unable-to-attach-or-mount-volumes\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.scmgalaxy.com\/tutorials\/"},{"@type":"ListItem","position":2,"name":"Kubernetes EKS Error: Unable to attach or mount volumes"}]},{"@type":"WebSite","@id":"https:\/\/www.scmgalaxy.com\/tutorials\/#website","url":"https:\/\/www.scmgalaxy.com\/tutorials\/","name":"scmGalaxy","description":"A Platform for Software Engineers","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.scmgalaxy.com\/tutorials\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.scmgalaxy.com\/tutorials\/#\/schema\/person\/a40e0dd38feee7c2779460a804d49361","name":"rajeshkumar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/82337e1fc8642c7f36795aa0775af055a4f38882994004b593efd31758d4434e?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/82337e1fc8642c7f36795aa0775af055a4f38882994004b593efd31758d4434e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/82337e1fc8642c7f36795aa0775af055a4f38882994004b593efd31758d4434e?s=96&d=mm&r=g","caption":"rajeshkumar"},"url":"https:\/\/www.scmgalaxy.com\/tutorials\/author\/rajeshkumar\/"}]}},"_links":{"self":[{"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/posts\/9950","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/comments?post=9950"}],"version-history":[{"count":5,"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/posts\/9950\/revisions"}],"predecessor-version":[{"id":9957,"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/posts\/9950\/revisions\/9957"}],"wp:attachment":[{"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/media?parent=9950"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/categories?post=9950"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.scmgalaxy.com\/tutorials\/wp-json\/wp\/v2\/tags?post=9950"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}