Hi there,
I have multiple target_groups for a NLB and I need to attach multiple instances to each target_group. The target_id being a string in aws_lb_target_group_attachment resource, I don’t see any easy way to achieve that. Whilst AWS console allows to select multiple instances during the target attachment why it’s not possible doing with terraform AWS provider?
// Get the instance ids of the NLB members
data "aws_instances" "nlb_insts" {
instance_tags = {
Name = "${var.vpc_names[var.idx]}${var.inst_role}0*"
}
instance_state_names = ["running", "stopped"]
}
// Creates the target-group
resource "aws_lb_target_group" "nlb_target_groups" {
count = length(var.nlb_listeners)
name = "nlb-tgr-${lookup(var.nlb_listeners[count.index], "target_port")}"
deregistration_delay = var.deregistration_delay
port = lookup(var.nlb_listeners[count.index], "target_port")
protocol = lookup(var.nlb_listeners[count.index], "protocol")
vpc_id = var.vpc_ids[var.idx]
health_check {
port = lookup(var.nlb_listeners[count.index], "health_port")
protocol = lookup(var.nlb_listeners[count.index], "protocol")
interval = var.health_check_interval
unhealthy_threshold = var.unhealthy_threshold
healthy_threshold = var.healthy_threshold
}
}
// Attach the target groups to the instance(s)
resource "aws_lb_target_group_attachment" "tgr_attachment" {
count = length(var.nlb_listeners)
target_group_arn = element(aws_lb_target_group.nlb_target_groups.*.arn, count.index)
target_id = [for sc in range(var.inst_count) : data.aws_instances.nlb_insts.ids[sc]]
port = lookup(var.nlb_listeners[count.index], "target_port")
}
Doing so, I get the obvious error:
Error: Incorrect attribute value type
on …/…/modules/elb/balencer.tf line 31, in resource “aws_lb_target_group_attachment” “tgr_attachment”:
31: target_id = [for sc in range(2) : data.aws_instances.nlb_insts.ids[sc]]
Inappropriate value for attribute “target_id”: string required.
Is there anything I’m missing here? If not, any suggestion(s) from anyone how to do that? Ant help would be greatly appreciated. I’m running out of ideas. thanks!
What you have here is a class of problem where you need to create something for every combination of two other things. In this case, you need a target group attachment for every combination of target group and instance.
The function setproduct is our main building-block here: given two or more sets or lists it will produce a flattened set or list representing each distinct combination of values from those input collections.
I’d also suggest using the resource for_each feature here, rather than count, so that you can add and remove both target groups and instances in the future without having to worry about disturbing the count indices.
Putting that all together, we get something like this:
data "aws_instances" "nlb_insts" {
instance_tags = {
Name = "${var.vpc_names[var.idx]}${var.inst_role}0*"
}
instance_state_names = ["running", "stopped"]
}
resource "aws_lb_target_group" "nlb_target_groups" {
for_each = {
# Target groups are identified by their target port
# numbers and protocols, which must be unique.
for l in var.nlb_listeners : "${l.protocol}:${l.target_port}"
}
name = "nlb-tgr-${each.value.target_port}"
deregistration_delay = var.deregistration_delay
port = each.value.target_port
protocol = each.value.protocol
vpc_id = var.vpc_ids[var.idx]
health_check {
port = each.value.health_port
protocol = each.value.protocol
interval = var.health_check_interval
unhealthy_threshold = var.unhealthy_threshold
healthy_threshold = var.healthy_threshold
}
}
resource "aws_lb_target_group_attachment" "tgr_attachment" {
for_each = {
for pair in setproduct(keys(aws_lb_target_group.nlb_target_groups), aws_instances.nlb_insts.ids) :
"${pair[0]}:${pair[1]}" => {
target_group = aws_lb_target_group.nlb_target_groups[pair[0]]
instance_id = pair[1]
}
}
target_group_arn = each.value.target_group.arn
target_id = each.value.instance_id
port = each.value.target_group.target_port
}
This should given you target groups with addresses like aws_lb_target_group.nlb_target_groups["TCP:80"] and attachments with addresses like aws_lb_target_group_attachment.tgr_attachment["TCP:80:i-abcd1234"].
One caveat here is that because I’m using the instance ids as part of the attachment keys they must be known during plan time. That works in this case because you’re retrieving pre-existing instance ids using a data source, but would not work if this configuration were the one responsible for creating those instances: Terraform wouldn’t know what those ids would be until after the plan is applied.
I’d usually recommend using keys that derive only from information that is statically-configured inside the current configuration to avoid that problem. In this case, that might mean using the instance indices as part of the key, which might still be subject to renumbering if any of the instances were to get replaced. That sort of problem is normally mitigated by using AWS autoscaling and letting it worry about creating these attachments, so that Terraform isn’t micro-managing each instance and its connections directly.
Thanks for the heads up, @apparentlymart and the for the code
I’m giving it a go now and will report back here.
Instances are created by a different module (but part of the same plan), so should be okay from that aspect. I do agree with your point about using ASG but this is not in the plan for this particular case. So had to find a way to deal with the individual instance(s) instead.
Looking at all of the heavy lifting, I still think that could be easily avoided if target_id could just simply accept a list instead of the string. I think that should be a valid type to support. Why it’s not possible?
I’m not sure of the details there (I wasn’t involved in the implementation of that resource type) but as a general rule the AWS provider team tends to follow the structure of the underlying AWS APIs because experience has shown that any higher-level abstraction invented by the AWS provider is subject to being broken by later changes to the AWS API, which are outside of the control of the AWS provider team.
With that said, I would guess that it’s designed the way it is because the underlying AWS API considers “Target Group Attachment” to be a relationship between one target group and one instance, and so by making it a list the provider would be mapping only one object in the Terraform configuration to many separate objects in the underlying API.
If you’d like to discuss the possibility of changing that interface, the best way to start that conversation would be to open a feature request issue. If you do decide to open one, please add a link to it here so any future readers of this topic can find it! (You may also wish to create the opposing link back from the issue to this topic, as some additional background context for the feature request.)
The Python boto3 also has the register_targets, where Targets also a list. So I think it’s well supported in the API layer.
I’ve happy to do a feature-request and will link it up here.
Thanks for opening that issue, @dsantanu. Given how that underlying API is designed, the goal was probably a little different: to allow the full set of attachments for a particular target group do be potentially managed across multiple separate resources. With the single resource taking a list, multiple aws_lb_target_group_attachment resources for the same target group would conflict with one another by overwriting each other’s lists.
With that said, we’ll see what the provider team has to say when they see your issue. They’ll certainly have a lot more context on this than I do!
Hi @apparentlymart,
I just got some time to try out the code you suggested and for the first for_each block I’m getting this error:
I tried my best to find a solution in the net but not much help there yet. Any idea what’s wrong here? Something to do with nlb_listeners variable declaration?
Any one knows what does the above error message mean? Don’t see any thing that useful anywhere including in the terraform documentation. Any help/pointer?
Sorry, it looks like I made an editing error when preparing that example. I intended to write this:
for_each = {
# Target groups are identified by their target port
# numbers and protocols, which must be unique.
for l in var.nlb_listeners : "${l.protocol}:${l.target_port}" => l
}
This creates a map from a generated key to an object representing the listener. The error message was complaining that the => clause was missing from the expression, and so Terraform thought that "${l.protocol}:${l.target_port}" was intended as the map value and the map key wasn’t specified at all.
thanks for the code @vsreelasya!
I was doing a bit of different stuff for a while but now back to this blancing-business again. I’ll give it a try shortly and report back.
It looks like this. You could do creating instance and NLB on same plan, make sure you pass the instance_id created to the NLB target group attachment if you are using instance_id or the private address of the instance. Hope I answered your question. Let me know if you have any other questions
Sorry for the delay in replying: I moved to do different things for a while.
Anyway, now I’m getting Error: Invalid for_each argument on the for_each for the target_group_attachment. As this post is several months old, I’ve posted it as a new topic here: Invalid for_each argument during the aws_lb_target_group_attachment
