Multiple Codepipeline within terraform - Duplicate key issue

Hey!

I have 2 modules codebuild and codepipeline and my objective is to create multiple pipelines through tfvars by creating arrays.

pipelines = [{
    name     = "artifact-mail-common",
    execution_mode = "QUEUED"
    triggers = { source_action_name = "mail_common", include_tags = ["feature/*", "main", "qw"] },
    stages = [
      { id = "mail-pl", name = "mail_common", category = "Source", owner = "AWS", provider = "CodeStarSourceConnection", output_artifacts = ["dummy_artifact"], namespaces = "SourceVariables", configuration = { FullRepositoryId = "dummy_repo1", BranchName = "main" } },
      { id = "mail-pl", name = "MyApprovalStage", category = "Approval", owner = "AWS", provider = "Manual",  configuration = { EnvironmentVariables = "" }},
      { id = "mail-pl", name = "devops", category = "Source", owner = "AWS", provider = "CodeStarSourceConnection", output_artifacts = ["devops_artifact"], configuration = { FullRepositoryId = "dummy_repo3", BranchName = "feature/devops" } },
      { id = "mail-pl", name = "java", category = "Build", owner = "AWS", provider = "CodeBuild", input_artifacts = ["dummy_artifact", "devops_artifact"], output_artifacts = ["javaArtifact"], namespaces = "JavaVariables", build_spec = "cd/java-dynamic-version-build.yaml", configuration = { EnvironmentVariables = "" }},
      { id = "mail-pl", name = "build", category = "Build", owner = "AWS", provider = "CodeBuild", input_artifacts = ["dummy_artifact", "devops_artifact"], output_artifacts = ["BuildArtifact"], namespaces = "BuildVariables", build_spec = "cd/java-build.yaml", configuration = { EnvironmentVariables = "EnvBuild.json.tmpl" } },
      { id = "mail-pl", name = "artifact-deploy", category = "Build", owner = "AWS", provider = "CodeBuild", input_artifacts = ["dummy_artifact", "devops_artifact", "BuildArtifact"], build_spec = "cd/java-artifact-deploy.yaml", configuration = { EnvironmentVariables = "EnvArtifactDeploy.json.tmpl" } },
    ]
  }, 
  {
    name     = "artifact-rest-api-auth-sso",
    execution_mode = "QUEUED" 
    triggers = { source_action_name = "api_authentication_sso", include_tags = ["feature/*", "main"] },
    stages = [
      { id = "auth-pl", name = "api_authentication_sso", category = "Source", owner = "AWS", provider = "CodeStarSourceConnection", output_artifacts = ["api_authentication_sso_artifact"], namespaces = "SourceVariables", configuration = { FullRepositoryId = "dummy_repo2", BranchName = "feature/asd" } },
      { id = "auth-pl", name = "MyApprovalStage-2", category = "Approval", owner = "AWS", provider = "Manual",  configuration = { EnvironmentVariables = "" }},
      { id = "auth-pl", name = "devops", category = "Source", owner = "AWS", provider = "CodeStarSourceConnection", output_artifacts = ["devops_artifact"], configuration = { FullRepositoryId = "dummy_repo3", BranchName = "feature/devops" } },
      { id = "auth-pl", name = "java", category = "Build", owner = "AWS", provider = "CodeBuild", creation = false, input_artifacts = ["api_authentication_sso_artifact", "devops_artifact"], output_artifacts = ["javaArtifact"], namespaces = "JavaVariables", build_spec = "cd/java-dynamic-version-build.yaml", configuration = { EnvironmentVariables = "" }},
      { id = "auth-pl", name = "build", category = "Build", owner = "AWS", provider = "CodeBuild", creation = false, input_artifacts = ["api_authentication_sso_artifact", "devops_artifact"], output_artifacts = ["BuildArtifact"], namespaces = "BuildVariables", build_spec = "cd/java-build.yaml", configuration = { EnvironmentVariables = "EnvBuild.json.tmpl" } },
      { id = "auth-pl", name = "artifact-deploy", category = "Build", owner = "AWS", provider = "CodeBuild", creation = false, input_artifacts = ["api_authentication_sso_artifact", "devops_artifact", "BuildArtifact"], build_spec = "cd/java-artifact-deploy.yaml", configuration = { EnvironmentVariables = "EnvArtifactDeploy.json.tmpl" } },
    ]
  },
]

This is an example tfvars array of pipelines which has 2 pipelines. I take the values of the map into a module through for_each and iterate each of them to their corresponding key inside the module. The issue occurs when I try to create for 2 pipelines where terraform gives me a

InvalidStageDeclarationException: Stage name 'java' is used more than once

This happens for all duplicate name’s. Logically this shouldn’t cause an issue as the stage’s are created within different pipelines, but since we are implementing through terraform, it considers this as duplicate elements and gives out an error.

Can anybody provide a viable solution for such a scenario?

For reference I will attach the module code. Please ignore values within tfvars as all are dummy. The code works when creating a single pipeline, the issue occurs when multiple pipelines are added within the pipelines array.

module "codepipeline" {
  source = "../modules/codepipeline"

  for_each = { for pipeline in var.pipelines : pipeline.name => pipeline }

  name               = each.key
  project_name       = var.name
  project_id         = var.project_id
  role_arn           = data.aws_iam_role.admin.arn
  s3_bucket_name     = data.aws_s3_bucket.codepipeline.bucket
  stages             = flatten([for pipeline in var.pipelines : pipeline.stages])

  execution_mode     = each.value.execution_mode
  source_action_name = each.value.triggers.source_action_name
  include_branch     = each.value.triggers.include_branch
  include_tags       = each.value.triggers.include_tags

  codestarconn_name = var.connection_arn

  tags = local.tags

  depends_on = [module.codebuild]
}

Hi,
What is happening here is that you are passing all the aggregated stage values to each item in your pipeline. In this case, you have duplicated values for:

  • “devops”
  • “java”
  • “build”
  • “artifact-deploy”

This is something you can confirm via console using the command:

flatten([for pipeline in local.pipelines : pipeline.stages.*.name])

Would each.value.stages work for you in this case? How does your module expect to receive the values? As a map?

Hey Wendel,

each.value.stages actually did work. Can’t for the world of me understand how I missed that. Thanks for the reply!

On a sidenote, is this the best way to implement the architecture for codepipeline? I checked online for any materials which shows the proper way but couldn’t seem to find any that shows how you can deploy multiple pipelines simultaneously using terraform.
Let me know if you have aby inputs on that.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.