For_each depends_on

how does one do a depends_on a for_each resource?

depends_on = [“aws_ecr_repository.ecr”]

Hi @FernandoMiguel,

What you showed there, aside from the 0.11-style quoting of the resource address, is the correct syntax. Did you encounter a problem when you tried that?

what is the equivalent for 0.12?

    Warning: Quoted references are deprecated

  on .terraform/modules/ecr/ecr.tf line 30, in resource "aws_ecr_lifecycle_policy" "lifecycle_policy":
  30:   depends_on = ["aws_ecr_repository.ecr"]

In this context, references are expected literally rather than in quotes.
Terraform 0.11 and earlier required quotes, but quoted references are now
deprecated and will be removed in a future version of Terraform. Remove the
quotes surrounding this reference to silence this warning.

ok that helps … wasn’t showing before

resource "aws_ecr_repository" "ecr" {
  for_each = toset(var.ecr_name)
  name     = each.key
}

resource "aws_ecr_lifecycle_policy" "lifecycle_policy" {
  for_each   = toset(var.ecr_name)
  repository = each.key

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 30,
            "description": "Expire untaged images older than 1 days",
            "selection": {
                "tagStatus": "untagged",
                "countType": "sinceImagePushed",
                "countUnit": "days",
                "countNumber": ${var.untag_days_countNumber}
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF

  depends_on = [aws_ecr_repository.ecr]
}

resource "aws_ecr_repository_policy" "repository_policy" {
  for_each   = toset(var.ecr_name)
  repository = each.key

  policy = <<EOF
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                "arn:aws:iam::0123456789:root",
                ]
            },
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:DescribeImages",
                "ecr:GetAuthorizationToken",
                "ecr:GetDownloadUrlForLayer"
            ]
        }
    ]
}
EOF

#   depends_on = [aws_ecr_repository.ecr]
}
  + create

Terraform will perform the following actions:

  # module.ecr.aws_ecr_lifecycle_policy.lifecycle_policy["foobaar"] will be created
  + resource "aws_ecr_lifecycle_policy" "lifecycle_policy" {
      + id          = (known after apply)
      + policy      = jsonencode(
            {
              + rules = [
                  + {
                      + action       = {
                          + type = "expire"
                        }
                      + description  = "Expire untaged images older than 1 days"
                      + rulePriority = 30
                      + selection    = {
                          + countNumber = 1
                          + countType   = "sinceImagePushed"
                          + countUnit   = "days"
                          + tagStatus   = "untagged"
                        }
                    },
                ]
            }
        )
      + registry_id = (known after apply)
      + repository  = "foobaar"
    }

  # module.ecr.aws_ecr_repository.ecr["foobaar"] will be created
  + resource "aws_ecr_repository" "ecr" {
      + arn                  = (known after apply)
      + id                   = (known after apply)
      + image_tag_mutability = "MUTABLE"
      + name                 = "foobaar"
      + registry_id          = (known after apply)
      + repository_url       = (known after apply)
    }

  # module.ecr.aws_ecr_repository_policy.repository_policy["foobaar"] will be created
  + resource "aws_ecr_repository_policy" "repository_policy" {
      + id          = (known after apply)
      + policy      = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "ecr:BatchCheckLayerAvailability",
                          + "ecr:BatchGetImage",
                          + "ecr:DescribeImages",
                          + "ecr:GetAuthorizationToken",
                          + "ecr:GetDownloadUrlForLayer",
                        ]
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = [
                              + "arn:aws:iam::0123456789:root",
                            ]
                        }
                    },
                ]
              + Version   = "2008-10-17"
            }
        )
      + registry_id = (known after apply)
      + repository  = "foobaar"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

get’s the following error

module.ecr.aws_ecr_repository.ecr["foobaar"]: Creating...
module.ecr.aws_ecr_repository_policy.repository_policy["foobaar"]: Creating...
module.ecr.aws_ecr_repository.ecr["foobaar"]: Creation complete after 0s [id=foobaar]
module.ecr.aws_ecr_lifecycle_policy.lifecycle_policy["foobaar"]: Creating...
module.ecr.aws_ecr_lifecycle_policy.lifecycle_policy["foobaar"]: Creation complete after 1s [id=foobaar]

Error: Error creating ECR Repository Policy: RepositoryNotFoundException: The repository with name 'foobaar' does not exist in the registry with id '0123456789'

  on .terraform/modules/ecr/ecr.tf line 33, in resource "aws_ecr_repository_policy" "repository_policy":
  33: resource "aws_ecr_repository_policy" "repository_policy" {

something is failing to use implicit dependency on a resource.
so i’m trying to use explicit.

In the log we can see that the repository creation is completing before Terraform tries to create the lifecycle policy:

module.ecr.aws_ecr_repository.ecr["foobaar"]: Creation complete after 0s [id=foobaar]
module.ecr.aws_ecr_lifecycle_policy.lifecycle_policy["foobaar"]: Creating...

So it looks like the dependencies in the configuration are correct, but that the aws_ecr_repository implementation in the AWS provider is returning success before the repository has fully committed. If that is true, unfortunately I don’t think there’s any way to strengthen that dependency from within the Terraform language: the provider would need to make sure the repository is ready to use before indicating that the creation has completed successfully.

Although the error message is not exactly the same, the overall behavior here seems similar to the following issue in the AWS provider:

Since that was originally opened in 2017, I think it’s plausible that the exact error message text returned from the remote API could’ve changed in the meantime, because both your message and the one reported there both seem to be saying (in different words) that the policy refers to a non-existing repository.

Thing is, the moment you add an explicit depends_on it works as expected.
That is why I was trying to find the correct syntax for for_each

Looking again at your configuration, it does seem like it was missing a dependency edge, though the log you shared showed the operations happening in the correct order anyway so perhaps the problem is a mixture of Terraform dependency handling and eventual consistency weirdness.

A different way to tell Terraform about the dependency, rather than falling back on depends_on, would be to refer to the repository resource from your for_each:

resource "aws_ecr_lifecycle_policy" "lifecycle_policy" {
  for_each = aws_ecr_repository.ecr

  repository = each.eky
  policy = jsonencode(...)
}

As well as declaring that the lifecycle policy objects depend on the repository objects, this would also allow using each.value in those blocks to refer to the full objects representing the repository. That’s not necessary in this case because the two policies are not variable per repository anyway, but I’m sharing it just as a general pattern to follow when your intent is to say e.g. “create one repository policy per repository”, which seems to be the case here.

If you do want to use depends_on here then indeed the syntax would be to just refer to the resource as normal: dependencies happen prior to resource expansion, so dependencies are always between resource blocks, not between the individual instances of those resource blocks:

  depends_on = [aws_ecr_repository.ecr]

Just as with the for_each strategy I described earlier, the dependency here is that all of the policies depend on all of the repositories, rather than each individual policy depending only on its corresponding repository instance. In practice that distinction rarely matters, but I just wanted to point it out since the question was about how depends_on and for_each interact.

1 Like

Oh wow
For_each of the resource created, and not of the var that produces the array.
Never thought of that before.

Time to rewrite some code.

Cheers

1 Like

@apparentlymart I reaching out again here, I see that depends_on and for_each don’t work well together, may be I am missing something here .

Is there any ETA for the fix in Terraform to fix this ?

Sharing my situation here :slight_smile: to make easy to co relate

General Order of IAM provisioning
IAM Role Creation --> IAM managed Policy Creation --> IAM policy attachment to IAM Roles

resource "aws_iam_policy" "managed_policy" {
  for_each      = length(local.managed_policy_list) > 0 ? toset(local.managed_policy_list) : []

  name          = each.key
  path          = "/"
  description   = each.key
  policy        = templatefile(format("${path.module}/policies/${each.key}%s",".tmpl"),
  {
    sub_acc_id  = var.sub_acc_id,
    account_id  = local.account_id,
    environment = var.environment,
    product     = var.product
  })
}

resource "aws_iam_role_policy_attachment" "eks_asl_instance_policy_attach" {
  for_each    = { for pl in local.eks_asl_node_policy_map: "${pl.role}-${element(split("/",pl.policy),1)}"  => pl }
  role        = each.value.role
  policy_arn  = each.value.policy

  depends_on  = [ aws_iam_policy.managed_policy ]
}

resource "aws_iam_role" "eks_node_asl_instance_role" {
  for_each             = { for nr in local.flat_asl_map: "${nr.asl}" => nr }
  name                 = "${title(each.value.asl)}-Instance-Role"
  path                 = "/"
  max_session_duration = 28800 // 8 hours
  permissions_boundary = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:policy/permissions-boundaries"
  assume_role_policy   = templatefile("${path.module}/policies/assume-role-01.tmpl",
  {trusted_aws_services   = ["ec2.amazonaws.com"] })
  tags =  merge({
    "type" = "asl_eks_instance_role",
    "ads:custdataaccess" = "true",
    "eks_cluster" = each.value.eks_name,
    "Name" = each.value.asl,
  },
  each.value.bucket_list
  )
  lifecycle {
    create_before_destroy = true
  }
}

locals {

  eks_asl_node_instance_policy_arns = [
    data.aws_iam_policy.ais_system_logs_policy.arn,
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    aws_iam_policy.managed_policy["AdsKMSPolicy"].arn,
    aws_iam_policy.managed_policy["AdsS3ReadPolicy"].arn,
    aws_iam_policy.managed_policy["AdsASLActionPolicy"].arn,
  ]
  eks_asl_node_policy_map = flatten([
    for k, v in aws_iam_role.eks_node_asl_instance_role:[
    for policy in local.eks_asl_node_instance_policy_arns: {
      role   = v.name
      policy = policy
  }
  ]])
}

}

Error


  on ads-policy-attach.tf line 16, in resource "aws_iam_role_policy_attachment" "eks_asl_instance_policy_attach":
  16:   for_each    = { for pl in local.eks_asl_node_policy_map: "${pl.role}-${element(split("/",pl.policy),1)}"  => pl }

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Hi @smitjainsj,

This error seems to be only related to for_each and not related to depends_on. Specifically it is reporting that one of the values contributing to the keys in your map is something that the corresponding provider determines only at apply time, and therefore you can’t use it as part of the identifier for a resource instance.

I think the root problem here is that some of your elements of local.eks_asl_node_instance_policy_arns are arns populated dynamically from aws_iam_policy resources, and the AWS provider is implemented to populate the arn attribute only during apply. That means that your list of ARNs, at plan time, consists of three known strings and three unknown strings. You then try to use those strings in your for_each keys in the expression split("/",pl.policy), which returns an unknown value itself because one of its inputs is unknown.

The requirement to make this work would be to use only values that you’ve written statically in the configuration as part of the key. Based on what you’ve shared, I think that could mean using the name of each policy as the identifier, rather than its arn, because you’ve set name = each.key and therefore those names are defined statically (albeit indirectly) from local.managed_policy_list.

Thanks @apparentlymart

thanks for sharing your feedback and is exactly correct that local.eks_asl_node_instance_policy_arns is list of arns which will be generated during runtime and later the local variable local.eks_asl_node_policy_map consumes those arn to create a flatten list.

Can you suggest a better to way to generate managed policy and later consume them in various roles ?

I tried multiple permutation and have always end using -target flag to create managed policy first and re run the terraform to consume those policies.

It’s hard to give specific advice here because I don’t have your full problem space in my head, but thinking tactically about what you shared I might try to change you local values like this:

locals {

  eks_asl_node_instance_policy_arns = {
    logs           = data.aws_iam_policy.ais_system_logs_policy.arn,
    ssm_instance   = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
    eks_cni        = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    ads_kms        = aws_iam_policy.managed_policy["AdsKMSPolicy"].arn,
    ads_s3_read    = aws_iam_policy.managed_policy["AdsS3ReadPolicy"].arn,
    ads_asl_action = aws_iam_policy.managed_policy["AdsASLActionPolicy"].arn,
  ]
  eks_asl_node_policy_map = flatten([
    for rk, role in aws_iam_role.eks_node_asl_instance_role : [
      for pk, policy_arn in local.eks_asl_node_instance_policy_arns: {
        role_key   = rk
        role_name  = role.name
        policy_key = pk
        policy_arn = policy_arn
      }
    ]
  ])
}

Your local.eks_asl_node_policy_map will then all have role_key and policy_key values defined statically in the configuration, making them good candidates to use as part of for_each unique keys downstream, even though the policy_arn attributes might be dynamically chosen during apply.

@apparentlymart

Thanks and I really appreciate you taking time here .

I will try that out, however for the sake of understanding here is the problem statement.

  1. Expect to have a policy create resource (Single) to generate Managed Policies in IAM via TMPL files.There can be a variable local.managed_list which will contain all the name of files which are candidate for managed policy creation.
  2. Expect to have policy attach resource (several) which can attach managed policy (permutation of both AWS managed policy and created from point 1) via a variable local.role1_policy_list to a multiple IAM roles created via another iam create resource . That iam create resource creates certain type of IAM roles based on requirement from user inputs.
  3. I do have a general IAM create role resource where I run a for_each over inputs (below example) given by user to generate all the IAM roles.
    eg.
IAM role input 
input_role_map:
  role1: # Role Name
  - user1@example.com # Trusted User to be added
  - user2@example.com
  role2:
  - user3@example.com
  - user2@example.com
  role3:
  - user1@example.com
  - user4@example.com
  role4:
  - user1@example.com
  - user2@example.com

I also tried to break this in a folder structure as follows instead of following flat structure :-

  1. iam-assumable-roles
  2. iam-managed-policies
  3. iam-attach-policies
  4. others.

However, with this there was too much to and fro of variable inputs within folders, where I could not sustain this.

As the number of roles I am trying to create much higher and dynamic compared to any standard environment and the scale is also quite large where this terraform code should scale to several of AWS Accounts.

I hope you can share your experience with writing such complex Terraform Code or modules in a better way.

Thanks again for helping here.