Solution: Error getting credentials

Terraform Providers Kubernetes
1

I wanted to share this solution:

Problem
We have been using Terraform to build Kubernetes clusters using Amazon Elastic Kubernetes Service (EKS) and plagued with intermittent errors below when running terraform init, plan or apply. The errors are specific to using any of the providers:

  • hashicorp/kubernetes
  • gavinbunney/kubectl
  • hashicorp/helm

Here are some examples:

Error : Get “https://ED5369UF1274843BF38BU€DE11111111.gr7.us-east-I . eks . amazonaws . com/api/vl/namespaces/test”: getting credentials: exec: executable aws failed with exit code 1

Error : Kubernetes cluster unreachable: Get “https://ED5369UF1277853BF38BU2DE77777777.gr7.us-east-l.eks.amazonaws.com/versi
on”: getting credentials: exec: executable aws failed with exit code 1

Error : failed to create kubernetes rest client for read of resource: Get “https://3U2D2U6D6DUEFC3U6€BF6A3688888888.gr7.us-east-I.eks.amazonaws.com/api?timeout=32s”: getting credentials: exec: executable aws failed with exit code 1

Solution
The problem was having the wrong configuration for these providers, this is how to resolve the problem.

Use aws_eks_cluster_auth to retrieve a token, then use this token for your providers. This is what actually resolved the intermittent problems.

Were using Terraform Cloud and retrieve the host and certificate from the state file.

# obtain a cluster token for providers, tokens are short lived (15 minutes)
data "aws_eks_cluster_auth" "cluster_auth" {
  name = module.eks-cluster.eks_cluster.name
}

provider "kubernetes" {
  host                   = data.tfe_outputs.this.values.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(data.tfe_outputs.this.values.cluster_certificate_authority)
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
}

provider "kubectl" {
  host                   = data.tfe_outputs.this.values.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(data.tfe_outputs.this.values.cluster_certificate_authority)
  load_config_file       = false
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.tfe_outputs.this.values.eks_cluster_endpoint
    cluster_ca_certificate = base64decode(data.tfe_outputs.this.values.cluster_certificate_authority)
    token                  = data.aws_eks_cluster_auth.cluster_auth.token
  }
}
2 Likes
2

I have tried this with Terraform v1.5.6, but it didn’t work

data "aws_eks_cluster_auth" "cluster_auth" {
  name = module.eks.cluster_endpoint
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
}

provider "kubectl" {
  apply_retry_count      = 5
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  load_config_file       = false
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.cluster_auth.token
  }
}

Output

Plan: 3 to add, 0 to change, 0 to destroy.
module.ingres_nginx_external.module.ingress_nginx.helm_release.this[0]: Creating...
module.ingres_nginx_internal.module.ingress_nginx.helm_release.this[0]: Creating...
module.eks_blueprints_kubernetes_addons.module.aws_load_balancer_controller.helm_release.this[0]: Creating...
╷
│ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
│ 
│   with module.eks_blueprints_kubernetes_addons.module.aws_load_balancer_controller.helm_release.this[0],
│   on .terraform/modules/eks_blueprints_kubernetes_addons.aws_load_balancer_controller/main.tf line 9, in resource "helm_release" "this":
│    9: resource "helm_release" "this" {
│
3

After about 2 or 3 days of digging, this finally resolved my issue. Great solution!

5

This helped a lot, I was using exec (below) and getting an error from terraform cloud.
With data "aws_eks_cluster_auth" the error gone away.

# don't use this, use aws_eks_cluster_auth instead:
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.aws_eks_xxx.cluster_name]
  }