Module resources recreated on every run - dependency/ordering issue - HELP! :)

Hello there!

I have an issue that’s driving me insane! It’s a clear dependency/ordering issue but so far i haven’t found a way round it…i’ve attempted to force the ordering in the way i need without any luck.

High level:

I have 2 modules, one for creating a VPC (and associated resources such as subnets, NACLs etc.) and the other for creating a VPC endpoint (and associated resources such as policy, security group etc.).

I require the ability to create 1…n VPCs with 1…n sets of subnets in one module, then 1…n VPC Endpoints in the other module that of course needs the VPC config (VPC & Subnet IDs) to exist before creation. When i create a single VPC, everything works fine and nothing changes on reapply. However, when i add a second VPC, the VPC endpoint resources from the first VPC want to recreate stating their VPC ID is not known before apply.

I’ve stripped the 2 modules right back so it creates a barebones VPC with a single subnet, and a single VPC endpoint defined and i’m still hitting the same issue.

I know i could just move the VPC endpoint resources to the VPC module, however i’m looking at this as a last resort as we use the VPC endpoint module in numerous other places and version control it for any updates etc.

Few things to note:

  1. We use data sources (aws_vpc & aws_subnets) with tag filters in the VPC endpoint module to select the VPC and Subnets
  2. If i remove the depends_on from the vpc_endpoint module then i get an Error: no matching EC2 VPC found error
  3. I have stripped back the resources for the sake of troubleshooting, however we would have multiple other VPC modules (e.g. vpc_edmz) that would be concatenated and included in the VPC endpoint local map variable for input to the VPC endpoint module.
  4. There are many more resources in the VPC module, each having their own for_each logic based off the subnets input. I’m guessing these could potentially be affecting the ordering.
  5. On first apply i can successfully create more than 1 VPC without issue, it only seems to be a problem if you apply once, add an additional VPC definition, then attempt to apply a second time.

Here are the resources:

VPC Module:

module "vpc_idmz" {
  for_each    = { for vpc in var.vpc_idmz : vpc["name"] => vpc }
  source      = "git::https://xxx"
  kms_key_arn = local.kms_key_arn
  vpc = merge(each.value,
    {
      network_zone            = "idmz"
      create_internet_gateway = false
    }
  )
}

VPC Endpoint Module

module "vpc_endpoint" {
  for_each        = { for vpc_endpoint in local.vpc_endpoint_map : "${vpc_endpoint["service_name"]}-${vpc_endpoint["vpc_name"]}" => vpc_endpoint }
  source          = "git::https://xxx"
  vpc_tag_filters = { Name = module.vpc_idmz[each.value["vpc_name"]].name }
  #vpc_tag_filters = { network_zone = each.value["vpc_network_zone"], Name = each.value["vpc_name"] }
  hosted_zone_id  = null
  vpc_endpoint = {
    name                = each.value["service_name"]
    vpc_endpoint_type   = each.value["vpc_endpoint_type"]
    private_dns_enabled = each.value["private_dns_enabled"]
    service_name        = "com.amazonaws.${data.aws_region.current.name}.${each.value["service_name"]}"
    policy              = local.vpc_endpoint_policies_map[each.value["service_name"]]["policy"]
    dns_name            = null
    subnet_tag_filters  = each.value["subnet_tag_filters"]
    security_group_ids  = []
    route_table_ids     = each.value["route_table_ids"]
    ingress_cidr_blocks = each.value["ingress_cidr_blocks"]
  }
  depends_on = [module.vpc_idmz]
}

VPC Endpoint Module Data Sources

data "aws_vpc" "selected" {
  tags = var.vpc_tag_filters
}
data "aws_subnets" "selected" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.selected.id]
  }
  tags = var.vpc_endpoint["subnet_tag_filters"]
}

VPC Input Variable

variable "vpc_idmz" {
  type = list(object({
    name                   = optional(string)
    network_zone           = optional(string, "idmz")
    network_function       = optional(string)
    cidr_block             = optional(string)
    additional_cidr_blocks = optional(list(string), [])
    enable_dns_support     = optional(bool, true)
    enable_dns_hostnames   = optional(bool, true)
    tags                   = optional(map(string), {})
    subnets = list(object({
      name                             = string
      type                             = string
      cidr_blocks                      = list(string)
      create_nat_gateway               = optional(bool, false)
      create_route_to_internet_gateway = optional(bool, false)
      endpoints = optional(list(object({
        vpc_endpoint_type   = optional(string, "Interface")
        private_dns_enabled = optional(bool, true)
        service_name        = optional(string)
        dns_name            = optional(string)
        ingress_cidr_blocks = optional(list(string), [])
        resources           = optional(list(string), [])
      })), [])
      nacl = object({
        ingress_rules = optional(list(object({
          protocol    = string
          rule_number = number
          rule_action = string
          cidr_block  = string
          from_port   = number
          to_port     = number
          })), [{
          rule_number = 100
          protocol    = -1
          rule_action = "allow"
          cidr_block  = "0.0.0.0/0"
          from_port   = 0
          to_port     = 0
        }])
        egress_rules = optional(list(object({
          protocol    = string
          rule_number = number
          rule_action = string
          cidr_block  = string
          from_port   = number
          to_port     = number
          })), [{
          rule_number = 100
          protocol    = -1
          rule_action = "allow"
          cidr_block  = "0.0.0.0/0"
          from_port   = 0
          to_port     = 0
        }])
      })
      tags                                = optional(map(string), {})
    }))
    security_group = optional(object({
      ingress_rules = optional(list(object({
        description       = string
        from_port         = number
        to_port           = number
        protocol          = string
        cidr_block        = string
        security_group_id = string
      })), [])
      egress_rules = optional(list(object({
        description       = string
        from_port         = number
        to_port           = number
        protocol          = string
        cidr_block        = string
        security_group_id = string
      })), [])
    }), {})
  }))
}

vpc_endpoint_map (flattened)

vpc_endpoint_map = flatten([
    for vpc in local.vpcs : [
      for subnet in vpc["subnets"] : [
        for vpc_endpoint in subnet["endpoints"] : {
          vpc_endpoint_type    = vpc_endpoint["vpc_endpoint_type"]
          private_dns_enabled  = vpc_endpoint["private_dns_enabled"]
          service_name         = vpc_endpoint["service_name"]
          resources            = vpc_endpoint["resources"]
          dns_name             = vpc_endpoint["dns_name"]
          subnet_tag_filters   = { type = subnet["type"], identifier = subnet["name"] }
          ingress_cidr_blocks  = vpc_endpoint["ingress_cidr_blocks"]
          vpc_id               = module.vpc_idmz[vpc["name"]].id
          vpc_name             = vpc["name"]
          vpc_network_zone     = vpc["network_zone"]
          vpc_network_function = vpc["network_function"]
          subnet_name          = subnet["name"]
          route_table_ids      = module.vpc_idmz[vpc["name"]].route_table_ids
        }
      ]
    ]
  ])

Now, the first run creates fine with the following inputs:

vpc_idmz = [
  {
    name         = "vpc-1"
    cidr_block   = "10.32.0.0/16"
    subnets = [
      {
        name                             = "subnets-1"
        type                             = "private"
        cidr_blocks                      = ["10.32.4.0/22"]
        create_nat_gateway               = false
        create_route_to_internet_gateway = false
        endpoints = [
          {
            service_name = "ec2"
          }
        ]
        nacl = {
          ingress_rules = []
          egress_rules = []
        }
      }
    ]
  }
]

…however, when i run the second time with an additional VPC object defined in the vpc_idmz input, all the VPC endpoint resources (endpoint, policy, security group) for the endpoint in the first VPC want to recreate.

New VPC input with second VPC defined:

vpc_idmz = [
  {
    name         = "vpc-1"
    cidr_block   = "10.32.0.0/16"
    subnets = [
      {
        name                             = "vpc-subnet-1"
        type                             = "private"
        cidr_blocks                      = ["10.32.4.0/22"]
        create_nat_gateway               = false
        create_route_to_internet_gateway = false
        endpoints = [
          {
            service_name = "ec2"
          }
        ]
        nacl = {
          ingress_rules = []
          egress_rules = []
        }
      }
    ]
  },
  {
    name         = "vpc-2"
    cidr_block   = "10.38.0.0/16"
    subnets = [
      {
        name                             = "vpc-subnet-2"
        type                             = "private"
        cidr_blocks                      = ["10.38.4.0/22"]
        create_nat_gateway               = false
        create_route_to_internet_gateway = false
        endpoints = [
          {
            service_name = "ec2"
          }
        ]
        nacl = {
          ingress_rules = [ ]
          egress_rules = []
        }
      }
    ]
  }

Resources showing recreation:

  # module.vpc_endpoint["ec2-vpc-1"].data.aws_vpc.selected will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_vpc" "selected" {
      + arn                                  = (known after apply)
      + cidr_block                           = (known after apply)
      + cidr_block_associations              = (known after apply)
      + default                              = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = (known after apply)
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = (known after apply)
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + state                                = (known after apply)
      + tags                                 = {
          + "Name" = "vpc-1"
        }

      + timeouts {
          + read = (known after apply)
        }
    }


  # module.vpc_endpoint["ec2-vpc-1"].aws_security_group.endpoint_security_group[0] must be replaced
-/+ resource "aws_security_group" "endpoint_security_group" {
      ~ arn                    = "arn:aws:ec2:eu-west-1:658084889611:security-group/sg-074d6ba2af5c6b996" -> (known after apply)
      ~ egress                 = [] -> (known after apply)
      ~ id                     = "sg-074d6ba2af5c6b996" -> (known after apply)
      ~ ingress                = [] -> (known after apply)
        name                   = "ec2-vpce"
      + name_prefix            = (known after apply)
      ~ owner_id               = "658084889611" -> (known after apply)
        tags                   = {
            "Name" = "ec2-access"
        }
      ~ vpc_id                 = "vpc-0e9ab532cee74b900" -> (known after apply) # forces replacement
        # (3 unchanged attributes hidden)
    }

So it looks like the aws_vpc data source in the VPC Endpoint module is not being evaluated at the correct time in order for the above to work. My question is…how can i correct this?

I’ve tried everything from using different depends_on, adding a data source to the level this is being applied from and trying to force a dependency loop, outputting the inputs from the VPC modules and using those in the local variable to force the correct ordering…no luck with anything!

Apologies for the long post!

Hi @lambbuster,

The key part of the output here is the reason for the data source being deferred: depends on a resource or a module with changes pending. This means you have told Terraform that the data source cannot be used if there are any changes pending in its dependencies, which according to the configuration, includes everything within module.vpc_idmz.

Adding depends_on should be done judiciously only when it’s required, and removing it from the vpc_endpoint module call will probably help here.

@jbardin - thank you for your reply. I totally agree that this is the issue, and i’ve normally been able to resolve similar situations.

However, if i remove the depends_on from the vpc_endpoint module, i get an error from the aws_vpc data source in question stating: Error: no matching EC2 VPC found…meaning without the dependency the data source is attempting to be evaluated before the VPC is created.

Is there a different way i can ensure the ordering without the depends_on ?

Thanks,
Phil

So it sounds like the vpc names are not derived from computed values of actual vpc resources, which means that Terraform cannot ensure the correct ordering without the depends_on given the current configuration.

To avoid this type of error, you should not have a managed resource and a data source representing the same logical resource within the same configuration. Because the vpc is being created within the same configuration, the data from that vpc should be fed directly into the module, rather than relying on the data source to lookup the vpc again.

If for some reason the overall structure of the modules cannot be fixed, the workaround would be to ensure that a computed value from the vpc managed resource is fed into the vpc_endpoint module, and have the aws_vpc data source explicitly depend only on that value.

@jbardin - thank you, this solved the issue! I was so close with troubleshooting but missed one small part of the module data source. Closing this gap and removing the depends_on at module level fixed the ordering. Cheers!