Need Help with Automating aws-auth ConfigMap Update in Terraform for EKS

razl · June 14, 2024, 11:12am

Hi everyone,

I’ve been encountering a persistent issue for a few days now and could really use some guidance. I have a set of Terraform files that are responsible for setting up an EKS cluster, including VPCs, roles, policies, and other configurations. Additionally, I have a GitHub Actions workflow that interacts with this setup to deploy various Kubernetes resources.

My challenge lies with integrating a manually created IAM role and policy (with OIDC) into the aws-auth ConfigMap using Terraform. When I update aws-auth manually, everything works perfectly. However, when I attempt to automate this update through Terraform during the initial setup, it leads to failures. Specifically, if I include the update in the node group creation, Terraform errors out, stating that the aws-auth ConfigMap already exists.

Here’s what I’ve tried:

Applying the aws-auth update manually — this works without issues.
Automating the update through Terraform, which fails during the node group creation or reports that aws-auth already exists if I add a dependency on the node group creation.

What could I be doing wrong, and what are the best practices for successfully automating this process?

I’ve already searched for solutions but haven’t found anything conclusive.

Thank you in advance for your help!

mcmillan.matthew · June 25, 2024, 8:43pm

I have just run into the same problem. Have you figured out a solution yet? I can’t even manually update aws-auth because I’m creating my EKS cluster with Terraform Cloud and the cluster only has the permissions for the assumed role that Terraform Cloud used to create the cluster.

I found this discussion from 2 years ago. It has a some options but none of them seem great.

github.com/terraform-aws-modules/terraform-aws-eks

How do I update aws-auth without using kubectl patch?

opened 07:26PM - 20 Jan 22 UTC

closed 09:48AM - 26 Jan 22 UTC

soostdijck

question

## Description I'm looking for a way to use the kubernetes provider to update… aws-auth. I need to add some roles to allow users to access the cluster via the console and a few other things. However the TF kubernetes provider doesn't support any way that I know of to update aws-auth after creation. I see examples using `kubectl patch` but I'm deploying using terraform cloud which is giving me issues TF cloud means that the cluster is created via a role that is assumed by the TF cloud workspace. These TF cloud instances are very limited they do not contain anything besides terraform, there is no kubectl, awscli or shell present and I can't add it. How can I patch aws-auth using the kubernetes provider? ## Versions - Terraform: 1.1.3 - Provider(s): kubernetes - Module: ## Reproduction Steps to reproduce the behavior: Use a TF cloud workspace to create the EKS cluster, then try to update `aws-auth` after the cluster is created. ### Code Snippet to Reproduce ``` locals { aws_auth_configmap_hcl = yamldecode(module.eks.aws_auth_configmap_yaml) } resource "kubernetes_config_map" "aws_auth" { metadata { name = "aws-auth" namespace = "kube-system" labels = merge( { "app.kubernetes.io/managed-by" = "Terraform" } ) } data = { mapRoles = jsonencode( concat( [ { rolearn = local.aws_user_role username = "LZAdministratorsRole" groups = [ "system:masters", ] }, { rolearn = local.aws_terraform_role username = "LZTerraformRole" groups = [ "system:masters", ] } ], yamldecode(local.aws_auth_configmap_hcl.data.mapRoles) ) ) } depends_on = [module.eks] } ``` ## Actual behavior ``` Error: configmaps "aws-auth" already exists with module.main.kubernetes_config_map.aws_auth on ../auth.tf line 9, in resource "kubernetes_config_map" "aws_auth": resource "kubernetes_config_map" "aws_auth" { ```

mcmillan.matthew · June 26, 2024, 5:43pm

This example code solved it for me:

github.com

jamiemo/k8s-eks-aws-auth-configmap/blob/main/main.tf

# 1. Find cluster based on tags
data "aws_resourcegroupstaggingapi_resources" "cluster" {
  resource_type_filters = ["eks:cluster"]

  tag_filter {
    key    = "Environment"
    values = ["dev"]
  }

  tag_filter {
    key    = "Project"
    values = ["jamiemo"]
  }
}

# 2. Extract cluster name
locals {
  cluster_arn      = data.aws_resourcegroupstaggingapi_resources.cluster.resource_tag_mapping_list[0].resource_arn
  cluster_name     = regex("^arn:aws:eks:.+:\\d+:cluster\\/(.+)$", local.cluster_arn)[0] 
}

This file has been truncated. show original

It uses the kubernetes_config_map_v1_data resource and concats the existing aws-auth config map with your new one.

Topic		Replies	Views
How to update aws-auth configmap using terraform for eks cluster AWS	0	127	June 13, 2024
AWS EKS - Terraform: Error configmap "aws-auth" does not exist AWS	14	7285	July 7, 2023
How can I modify the aws-auth configmap created by aws_eks_cluster AWS	0	623	November 2, 2020
The configmap “aws-auth” does not exist AWS k8s	6	1156	August 11, 2023
Is is possible to automate the import of resources that are created by other resources? AWS	0	559	November 2, 2020

Need Help with Automating aws-auth ConfigMap Update in Terraform for EKS

Related topics