Issue while registering targets for alb target group

Terraform Providers AWS
1

I am trying to create a vpc interface endpoint for s3 and an alb in the same module. Here is my code below-:

resource "aws_vpc_endpoint" "s3"{
    vpc_id = var.vpc_id
    service_name = "com.amazonaws.eu-west-3.s3"
    vpc_endpoint_type = "Interface"

    security_group_ids = [
        aws_security_group.allow_all_traffic.id
    ]

    subnet_ids = var.subnet_ids

    private_dns_enabled = true
}

resource "aws_lb_target_group_attachment" "test_0" {
 
  depends_on = [ aws_vpc_endpoint.s3 ]
  for_each         = aws_vpc_endpoint.s3.network_interface_ids
  target_group_arn = aws_lb_target_group.ip-example.arn
  target_id        = each.value.private_ip
  port             = 443
 
}

I am getting the error specified below-:

│ The "for_each" set includes values derived from resource attributes that cannot be determined until apply, and so Terraform 
│ cannot determine the full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to use a map value where the keys are defined statically in your  
│ configuration and where only the values contain apply-time results.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends  
│ on, and then apply a second time to fully converge.

I have read many posts but i have not been able to find any workaround yet.

2

Hi @sparshnarang22,

Unfortunately I think the design of the underlying EC2 API causes a problem here.

From what I understand from the documentation about VPC Endpoints, EC2 allocates one network interface for each of the subnets you specify in subnet_ids. Therefore in principle we could derive a map from subnet id to network interface id and use that in for_each.

However, the underlying API only returns an unordered set of network interface ids, without any information about which network interface belongs to which subnet and without any guarantee that the interfaces will stay in a predictable order on future changes.

Therefore I think unfortunately the most robust solution would be to perform the creation of this particular configuration in two steps:

  • terraform apply -target=aws_vpc_endpoint.s3 to get everything up to the VPC endpoint created
  • terraform apply as normal to get everything else created

Then you can use terraform apply in the normal way for your ongoing maintenance as long as you don’t change the subnet ids associated with your VPC endpoint, or make any other changes that would cause the VPC endpoint to be replaced.

There is a potential alternative option that is slightly risky: it could get these objects created all in one round but might lead to some unwanted spurious updates if you change var.subnet_ids in future.

The general idea here is to rely on the fact that the number of elements in aws_vpc_endpoint.s3.network_interface_ids should always match the number of elements in var.subnet_ids, assuming I’m understanding the EC2 documentation correctly.

The first part of the trick is to use the sort function to convert the set of network interface IDs into a list of network interface IDs, sorted lexically:

locals {
  network_interface_ids_sorted = sort(aws_vpc_endpoint.s3.network_interface_ids)
}

If we assume that this list is always the same length as var.subnet_ids, then we can assume that any index that would be valid for var.subnet_ids is valid for local.network_interface_ids_sorted too, and thus write something like this:

resource "aws_lb_target_group_attachment" "test_0" {
  count = length(var.subnet_ids)

  target_group_arn = aws_lb_target_group.ip-example.arn
  target_id        = local.network_interface_ids_sorted[count.index]
  port             = 443
}

Because the network interface IDs are in no particular order, we cannot assume that a particular count.index will stay associated with a particular network interface ID over time.

This means that if you change var.subnet_ids in future – thereby also causing the network interface ids to change – the correlations between instance indices and network interfaces might change, causing Terraform to propose to change the network interface ids associated with some or all of the existing instances.

If you decide to use this strategy then I strongly suggest rehearsing what happens when var.subnet_ids changes to make sure the result is acceptable. There are a few different things that could happen which might be undesirable depending on your requirements:

  • If the set of network interface ids changes substantially enough, there might be a brief period during the apply phase where all target group attachments are being updated at once. I don’t know what effect that would have for those attempting to access your service via this load balancer; it might cause temporary downtime.
  • If the load balancing service has a rule that the target_id must be unique across all attachments for a particular target group then Terraform might encounter errors trying to reassign the target ids. For example, if you remove an element from var.subnet_ids that then causes one of the network interface ids to be removed, all of the network interfaces that sort after it in the list are likely to get updated to point to different target ids. Terraform will perform those updates in an unpredictable order because it doesn’t expect these actions to be interdependent, and so the remote API might return an error.
  • Depending on exactly how the provider represents the fact that changes to subnet_ids cause changes to network_interface_ids, you might find that the whole set of network interface ids becomes unknown again each time the subnet ids change, which will appear as Terraform proposing to set all of the target_id arguments to (known after apply).

It’ll be up to you to decide if whatever behavior you observe is acceptable for the way your module will be used. If you’re not sure then I would suggest returning to my original suggestion of applying this change in two steps whenever the set of network interface ids is changing.

3

Hi @apparentlymart thanks for the quick response! I don’t think i made myself completely clear with the topic i typed but this is how i overcame the issue of having unknown keys at apply time-

resource “aws_lb_target_group” “ip-example” {
name = “tf-example-lb-tg”
port = 443
protocol = “HTTPS”
target_type = “ip”
vpc_id = var.vpc_id

health_check {
enabled = true
protocol = “HTTP”
port = 80
matcher = “200,307,405”
}

depends_on = [ aws_vpc_endpoint.s3 ]
}

locals {
list_network_interface_ids = tolist(aws_vpc_endpoint.s3.network_interface_ids)
subnet_to_network_interface_map = {
for idx, subnet_id in var.subnet_ids : idx => local.list_network_interface_ids[idx]
}
}

data “aws_network_interface” “example”{
for_each = local.subnet_to_network_interface_map
id = each.value
}

resource “aws_lb_target_group_attachment” “test_0” {

depends_on = [ aws_vpc_endpoint.s3 ]
for_each = data.aws_network_interface.example
target_group_arn = aws_lb_target_group.ip-example.arn
target_id = each.value.private_ip
port = 443

}

Basically i made a map with keys as subnet ids and values as network interface ids and then registered their private ips with the target group by fetching their data.

4

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.