Hello there!
I have an issue that’s driving me insane! It’s a clear dependency/ordering issue but so far i haven’t found a way round it…i’ve attempted to force the ordering in the way i need without any luck.
High level:
I have 2 modules, one for creating a VPC (and associated resources such as subnets, NACLs etc.) and the other for creating a VPC endpoint (and associated resources such as policy, security group etc.).
I require the ability to create 1…n VPCs with 1…n sets of subnets in one module, then 1…n VPC Endpoints in the other module that of course needs the VPC config (VPC & Subnet IDs) to exist before creation. When i create a single VPC, everything works fine and nothing changes on reapply. However, when i add a second VPC, the VPC endpoint resources from the first VPC want to recreate stating their VPC ID is not known before apply.
I’ve stripped the 2 modules right back so it creates a barebones VPC with a single subnet, and a single VPC endpoint defined and i’m still hitting the same issue.
I know i could just move the VPC endpoint resources to the VPC module, however i’m looking at this as a last resort as we use the VPC endpoint module in numerous other places and version control it for any updates etc.
Few things to note:
- We use data sources (aws_vpc & aws_subnets) with tag filters in the VPC endpoint module to select the VPC and Subnets
- If i remove the
depends_on
from thevpc_endpoint
module then i get anError: no matching EC2 VPC found
error - I have stripped back the resources for the sake of troubleshooting, however we would have multiple other VPC modules (e.g.
vpc_edmz
) that would be concatenated and included in the VPC endpoint local map variable for input to the VPC endpoint module. - There are many more resources in the VPC module, each having their own
for_each
logic based off the subnets input. I’m guessing these could potentially be affecting the ordering. - On first apply i can successfully create more than 1 VPC without issue, it only seems to be a problem if you apply once, add an additional VPC definition, then attempt to apply a second time.
Here are the resources:
VPC Module:
module "vpc_idmz" {
for_each = { for vpc in var.vpc_idmz : vpc["name"] => vpc }
source = "git::https://xxx"
kms_key_arn = local.kms_key_arn
vpc = merge(each.value,
{
network_zone = "idmz"
create_internet_gateway = false
}
)
}
VPC Endpoint Module
module "vpc_endpoint" {
for_each = { for vpc_endpoint in local.vpc_endpoint_map : "${vpc_endpoint["service_name"]}-${vpc_endpoint["vpc_name"]}" => vpc_endpoint }
source = "git::https://xxx"
vpc_tag_filters = { Name = module.vpc_idmz[each.value["vpc_name"]].name }
#vpc_tag_filters = { network_zone = each.value["vpc_network_zone"], Name = each.value["vpc_name"] }
hosted_zone_id = null
vpc_endpoint = {
name = each.value["service_name"]
vpc_endpoint_type = each.value["vpc_endpoint_type"]
private_dns_enabled = each.value["private_dns_enabled"]
service_name = "com.amazonaws.${data.aws_region.current.name}.${each.value["service_name"]}"
policy = local.vpc_endpoint_policies_map[each.value["service_name"]]["policy"]
dns_name = null
subnet_tag_filters = each.value["subnet_tag_filters"]
security_group_ids = []
route_table_ids = each.value["route_table_ids"]
ingress_cidr_blocks = each.value["ingress_cidr_blocks"]
}
depends_on = [module.vpc_idmz]
}
VPC Endpoint Module Data Sources
data "aws_vpc" "selected" {
tags = var.vpc_tag_filters
}
data "aws_subnets" "selected" {
filter {
name = "vpc-id"
values = [data.aws_vpc.selected.id]
}
tags = var.vpc_endpoint["subnet_tag_filters"]
}
VPC Input Variable
variable "vpc_idmz" {
type = list(object({
name = optional(string)
network_zone = optional(string, "idmz")
network_function = optional(string)
cidr_block = optional(string)
additional_cidr_blocks = optional(list(string), [])
enable_dns_support = optional(bool, true)
enable_dns_hostnames = optional(bool, true)
tags = optional(map(string), {})
subnets = list(object({
name = string
type = string
cidr_blocks = list(string)
create_nat_gateway = optional(bool, false)
create_route_to_internet_gateway = optional(bool, false)
endpoints = optional(list(object({
vpc_endpoint_type = optional(string, "Interface")
private_dns_enabled = optional(bool, true)
service_name = optional(string)
dns_name = optional(string)
ingress_cidr_blocks = optional(list(string), [])
resources = optional(list(string), [])
})), [])
nacl = object({
ingress_rules = optional(list(object({
protocol = string
rule_number = number
rule_action = string
cidr_block = string
from_port = number
to_port = number
})), [{
rule_number = 100
protocol = -1
rule_action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 0
to_port = 0
}])
egress_rules = optional(list(object({
protocol = string
rule_number = number
rule_action = string
cidr_block = string
from_port = number
to_port = number
})), [{
rule_number = 100
protocol = -1
rule_action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 0
to_port = 0
}])
})
tags = optional(map(string), {})
}))
security_group = optional(object({
ingress_rules = optional(list(object({
description = string
from_port = number
to_port = number
protocol = string
cidr_block = string
security_group_id = string
})), [])
egress_rules = optional(list(object({
description = string
from_port = number
to_port = number
protocol = string
cidr_block = string
security_group_id = string
})), [])
}), {})
}))
}
vpc_endpoint_map (flattened)
vpc_endpoint_map = flatten([
for vpc in local.vpcs : [
for subnet in vpc["subnets"] : [
for vpc_endpoint in subnet["endpoints"] : {
vpc_endpoint_type = vpc_endpoint["vpc_endpoint_type"]
private_dns_enabled = vpc_endpoint["private_dns_enabled"]
service_name = vpc_endpoint["service_name"]
resources = vpc_endpoint["resources"]
dns_name = vpc_endpoint["dns_name"]
subnet_tag_filters = { type = subnet["type"], identifier = subnet["name"] }
ingress_cidr_blocks = vpc_endpoint["ingress_cidr_blocks"]
vpc_id = module.vpc_idmz[vpc["name"]].id
vpc_name = vpc["name"]
vpc_network_zone = vpc["network_zone"]
vpc_network_function = vpc["network_function"]
subnet_name = subnet["name"]
route_table_ids = module.vpc_idmz[vpc["name"]].route_table_ids
}
]
]
])
Now, the first run creates fine with the following inputs:
vpc_idmz = [
{
name = "vpc-1"
cidr_block = "10.32.0.0/16"
subnets = [
{
name = "subnets-1"
type = "private"
cidr_blocks = ["10.32.4.0/22"]
create_nat_gateway = false
create_route_to_internet_gateway = false
endpoints = [
{
service_name = "ec2"
}
]
nacl = {
ingress_rules = []
egress_rules = []
}
}
]
}
]
…however, when i run the second time with an additional VPC object defined in the vpc_idmz
input, all the VPC endpoint resources (endpoint, policy, security group) for the endpoint in the first VPC want to recreate.
New VPC input with second VPC defined:
vpc_idmz = [
{
name = "vpc-1"
cidr_block = "10.32.0.0/16"
subnets = [
{
name = "vpc-subnet-1"
type = "private"
cidr_blocks = ["10.32.4.0/22"]
create_nat_gateway = false
create_route_to_internet_gateway = false
endpoints = [
{
service_name = "ec2"
}
]
nacl = {
ingress_rules = []
egress_rules = []
}
}
]
},
{
name = "vpc-2"
cidr_block = "10.38.0.0/16"
subnets = [
{
name = "vpc-subnet-2"
type = "private"
cidr_blocks = ["10.38.4.0/22"]
create_nat_gateway = false
create_route_to_internet_gateway = false
endpoints = [
{
service_name = "ec2"
}
]
nacl = {
ingress_rules = [ ]
egress_rules = []
}
}
]
}
Resources showing recreation:
# module.vpc_endpoint["ec2-vpc-1"].data.aws_vpc.selected will be read during apply
# (depends on a resource or a module with changes pending)
<= data "aws_vpc" "selected" {
+ arn = (known after apply)
+ cidr_block = (known after apply)
+ cidr_block_associations = (known after apply)
+ default = (known after apply)
+ dhcp_options_id = (known after apply)
+ enable_dns_hostnames = (known after apply)
+ enable_dns_support = (known after apply)
+ enable_network_address_usage_metrics = (known after apply)
+ id = (known after apply)
+ instance_tenancy = (known after apply)
+ ipv6_association_id = (known after apply)
+ ipv6_cidr_block = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ state = (known after apply)
+ tags = {
+ "Name" = "vpc-1"
}
+ timeouts {
+ read = (known after apply)
}
}
# module.vpc_endpoint["ec2-vpc-1"].aws_security_group.endpoint_security_group[0] must be replaced
-/+ resource "aws_security_group" "endpoint_security_group" {
~ arn = "arn:aws:ec2:eu-west-1:658084889611:security-group/sg-074d6ba2af5c6b996" -> (known after apply)
~ egress = [] -> (known after apply)
~ id = "sg-074d6ba2af5c6b996" -> (known after apply)
~ ingress = [] -> (known after apply)
name = "ec2-vpce"
+ name_prefix = (known after apply)
~ owner_id = "658084889611" -> (known after apply)
tags = {
"Name" = "ec2-access"
}
~ vpc_id = "vpc-0e9ab532cee74b900" -> (known after apply) # forces replacement
# (3 unchanged attributes hidden)
}
So it looks like the aws_vpc
data source in the VPC Endpoint module is not being evaluated at the correct time in order for the above to work. My question is…how can i correct this?
I’ve tried everything from using different depends_on
, adding a data source to the level this is being applied from and trying to force a dependency loop, outputting the inputs from the VPC modules and using those in the local variable to force the correct ordering…no luck with anything!
Apologies for the long post!