Hello all,
in our organization, we use Service Catalog products in the AWS Service Catalog service in order to create kind-of managed EC2 instances. The Service Catalog product will launch an EC2 instance alongside with some other resources and will do some management stuff around the EC2 instance. Then, the provisioned product will return the Instance Id of the created EC2 instance as Output.
Then, we use a Terraform module as a wrapper for Terraform users to create such a Service Catalog provisioned product - I’ll call it VM module starting now. The Terraform module will take the output and will pass it to its own outputs and finally presents the Instance Id also to other resources that are created with Terraform.
One case of such resources using the Instance Id are Load Balancers, that we also create and manage in Terraform.
A Load Balancer will take the Instance Id from the module output and will use it as input for the Load Balancer Target Group Attachment (aws_lb_target_group_attachment resource in TF).
And this is where our issue starts:
Everytime, an update is done on the Service Catalog Provisioned Product, the outputs of that the VM module are “known after apply”, including the Instance Id. I guess that’s actually expected, as Terraform cannot evaluate what’s happening on the CloudFormation stack update that is behind a Service Catalog product update.
But in turn, also the Load Balancer Target Group Attachment, will show a change on its “target_id” attribute, as the target Instance Id might change.
What’s happening then is a bit strange for me:
The Load Balancer will deregister the target first. It will then update the Provisioned product and will then register the target again.
It does not wait, if the output has actually changed (because in most cases of our module it would not change). This means that actually every update applied on the VM module will result in a detach/re-attach of the EC2 from the Load Balancer, which includes a downtime as the VM is not connected to the Load Balancer for a some minutes.
I have also tested to use “create_before_destroy” as lifecycle on the Load Balancer target group attachment resource in Terraform. This did not work as well, with this behaviour:
With this lifecycle, Terraform also shows “known after apply” for the outputs in the plan and displays a change in the target_id for the LB Target Group Attachment. But this time it will first update the VM module. This means, that the “aws_lb_target_group_attachment” that is updated after the VM module should know now, if the Instance Id as target_id has changed or not. And even if the Instance ID output did not change on the module update, it will still try to register the same Instance Id as target again. This returns success with nothing changed on the Target Group, as the target is already attached. But then, due to create_before_destroy, it will destory the previous attachment and will therefore deregister the previous target (which is the same than the “new” target) and the LB will end up with no Instance being registered.
We already opened a GitHub issue here: [Enhancement]: Output from ServiceCatalog execution should change after ServiceCatalog update ? · Issue #37219 · hashicorp/terraform-provider-aws · GitHub
But the ticket was closed without any advise (should be requested in Core repository) and it was also proposed to raise the situation here.
So maybe you have some suggestions for us to solve or workaround the situation.
I appreciate any hints!
Thanks,
Michael