Terraform issues with module dependency using depends_on Meta-Argument

Terraform
1

H there,

I’m facing a dependency issue, whilst using depends_on with for_each.

I have child module: subnet, which needs some information from aws_vpc_endpoint resource and is being called from the root module that natively hosts the aws_vpc_endpoint resource. So, I have this in my code:

// Subnets for TGW attachment only
module "tgw_subnets" {
  for_each     = toset(keys(var.vpc_list))
  ....
  ....
  vpc_gw_eps   = local.endpoints_gws
  depends_on   = [aws_vpc_endpoint.gw_eps]
}

where the use of vpc_gw_eps is throwing in some error that generated in subnet module:

Error: Invalid for_each argument

  on .terraform/modules/tgw_subnets/include/subnet/nacl.tf line 35, in resource "aws_network_acl_rule" "gw_eps":
  35:   for_each = {
  36:     for ep in local.gw_eps_nacl : "${ep.type}:${ep.rule}" => {
  37:       "cidr" = ep.cidr,
  38:       "rule" = ep.rule,
  39:       "type" = ep.type,
  40:     }
  41:   }

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

here is my local.endpoints_gws:

endpoints_gws = {
    for vpc in keys(var.vpc_list) : vpc => {
      for ep in var.vpc_endpoints_gw : ep => {
        "arn"            = aws_vpc_endpoint.gw_eps["${vpc}:${ep}"].arn,
        "cidr_blocks"    = aws_vpc_endpoint.gw_eps["${vpc}:${ep}"].cidr_blocks,
        "id"             = aws_vpc_endpoint.gw_eps["${vpc}:${ep}"].id,
        "prefix_list_id" = aws_vpc_endpoint.gw_eps["${vpc}:${ep}"].prefix_list_id,
        "vpc_id"         = aws_vpc_endpoint.gw_eps["${vpc}:${ep}"].vpc_id,
      }
    }
  }

which is being passed on to subnet module, being further processed thereafter, as the local variables:

gw_ep_cidrs = flatten([
    for ep in keys(var.vpc_gw_eps[var.vpc_name]) :
    var.vpc_gw_eps[var.vpc_name][ep].cidr_blocks
])

gw_eps_nacl = flatten([
    for type in ["egress", "ingress"] : [
      for idx, cdr in sort(local.gw_ep_cidrs) : {
        "cidr" = cdr,
        "rule" = 200 + (2 * idx),
        "type" = type,
      }
    ]
])

and then gw_eps_nacl is used in the for_each, where the error actually being generated. I understand why it’s an issue for subnet module but why module.tgw_subnets is not waiting until aws_vpc_endpoint.gw_eps is completed? Isn’t that explicit depends_on for that purpose?

Any idea what’s am I doing wrong? How can I make it working? I’m using
Terraform v0.13.6.

2

The for_each and count arguments expand a single resource definition into multiple instances. To be able to plan the sequence of operations, Terraform needs to know how many instances each resource will have before the apply process can begin.

As a result, it’s highly recommended only to use static values (e.g. from input variables) with for_each and count. While Terraform supports referencing other resources (either directly, or indirectly through locals), this can result in situations like this one.

The configuration you describe here currently cannot be applied by Terraform in a single run. The depends_on argument can help add implicit dependency edges to the Terraform graph to help order operations, but it cannot cause Terraform to perform a multi-step apply—it still needs to know the values of the for_each arguments to construct the plan.

One way forward here would be to use the -target argument with terraform apply to ensure that the aws_vpc_endpoint.gw_eps resources are created first, then run a subsequent terraform apply to create the rest of the infrastructure.

3

I ended up using -target in development env but the actually it destined for running as pipeline job, where target cannot be applied.

As I said, I do understand the issue with for_each itself, which makes perfect sense, but I still don’t really understand why the depends_on won’t help here. It should not even try to process the module "tgw_subnets until aws_vpc_endpoint.gw_eps is applied.

Where exactly that multi-step apply is/are happening? Would be great, if you have some explanation for me.

4

Manual depends_on allows Terraform to sequence operations like create or destroy within an apply, but all of those operations happen after the “expand” step. This means that anything required to “expand” a resource into multiple instances (i.e. for_each or count arguments) is not affected by the use of depends_on.

Multi-step/progressive apply for situations like this is something that we’d like Terraform to be able to do one day, but I don’t think there are any concrete plans to work on that at the moment.

1 Like
5

okay, thanks!
I have to see what else I can do to get around that issue.