Hello.
We have a terraform module with a few child modules within it.
One of the child modules contains the following resource and data source:
- resource.aws_instance
- data.template_cloudinit_config
- The resource.aws_instance is using the data.template_cloudinit_config rendered output as it’s user_data
- The data.template_cloudinit_config is looking at a simple bash script
I only want the aws_instance to redeploy when the bash script has changed
The issue I am having is that the aws_instance is being forced to be replaced every terraform apply.
This is due to terraform re-rendering the data.template_cloudinit_config despite no changes to the bash script it is tracking.
Is there any reason why this is happening?
This only started happening when I moved these resources into a child module, originally they were in the parent module and this never happened.
Hi @MattTDickinson,
To answer this properly I’d need to see the relevant configuration and the associated output from terraform plan, but based on what you’ve shared my first guess would be that your data block depends on a resource block that has changes planned, and so Terraform must wait until the apply phase to call that data source in case the managed resource change affects the outcome. (Terraform can’t tell that template_cloudinit_config doesn’t access any network services, so it needs to be conservative.)
If the data resource gets deferred to the apply phase then its result would be unknown during planning, and so Terraform cannot determine whether it has changed compared to the previous user_data value.
If my theory is correct then the answer would be to make sure the data block doesn’t depend on something that is changing. If this is happening in every plan then that suggests that you have an incorrectly-configured resource block that isn’t converging after a single plan and apply (sometimes known as a “permadiff”, in which case I would suggest fixing that as the root cause.
If the above is not helpful, please share a real configuration example and the plan output that shows the full set of unwanted changes.
I can’t post a lot due to confidentiality
resource.aws_instance
resource "aws_instance" "red" {
ami = var.ami_id
instance_type = var.instance_size
iam_instance_profile = var.instance_profile
key_name = var.key_pair_name
user_data = data.template_cloudinit_config.red.rendered
user_data_replace_on_change = true
ebs_optimized = true
root_block_device {
volume_type = "gp3"
}
volume_tags = var.additional_resource_tags
network_interface {
network_interface_id = aws_network_interface.red.id
device_index = 0
}
tags = var.resource_tags
lifecycle {
ignore_changes = [ebs_block_device]
}
}
data source
data "template_cloudinit_config" "red" {
gzip = true
base64_encode = true
part {
filename = "cloud-config.sh"
content_type = "text/x-shellscript"
content = local.part1
}
local
part1 = templatefile("${path.module}/scripts/cloud-config.sh", {
region = var.region
workflow = var.workflow
environment = var.environment
})
An example ofAn example of terraform action plan
# module.xyz.module.xyz["xyz"].data.template_cloudinit_config.red will be read during apply
# (config refers to values not yet known)
<= data "template_cloudinit_config" "red" {
~ id = "1878789531" -> (known after apply)
~ rendered = "H4sIAAAAAAAA/8RZ73LbOJL/zqq8QwdRVknKEEXJsh3NKnu5xNndus1kKp65ralMTgURLQljEuACoGTNxO9+BYCkKFlynLupu0rFIoFGoxv9B79uvlHSorT0x02BY8jLzIqCaRvn4gb5dzBTpeRMbybk/d/fX/77h5++f/v6488kcm/0P1EboeQYkl7/UfQoorRN9Ciqeb8VplBGWE/LrGXpMkdpv4O5yFCyHCckzVTJaarkXCx6Zkm2i3/UTJo5anopU8WFXIzhfCZsi8BLbvHGxjfULDHLTKpFYR9F70WOd4R88jieCRnPmFlG0RNY1dOj6MkTKDK2oYOLpA8USiPkAvI18gWCyJn7K41FxkHN/SPLMgh7megJ+K3TJabXwIVhswwnV28GSXJ2cvVm0L8IP+EtGZ36n4vE/wxeOkmavaN5KVN3WKBLOeUqvUY99QI8ew6/RwBPYIG2FolHEB6ngk86zwJ5GDHwBRYaC+gGLbrwBdj6Grq/F1pIC53hbfe552fQ1po6ewCTHPJ1qjSCKAwICXYpDKQqz5nkfokXdzg47QPjHDK1MGAVsLXxs5RmakG5FivUE7Y2fv6X1pQqLFTjVONCKDnBkq7RWDo4SphqZBbpQquymFhd4lHKQBIvdFFkrPqhSVzpeGyVsRpZPik0FlpxWi+js6xEGtbS8pqiWSKdxeGo79WMM4vWeeFc6ZzZSfeXT09n8JSfwNOf4enfxk/fj59e/fK56xlUptOlBEp9WDxMEL8YgFKNxjJtJyxbs40ByusZhPf/vHz718vp96/fX34jVwRjmS3N36wtrlCvUP+gtJ1c9Pv9Nvs3Hz5eTl+/ffvx8upqkvR7o0FvkPSS09OT6mXQS0b187A3SM7q1SuItVI29ttb5wBVGvjVKDmO0aaV2eLWeKOzRLtW+nqyVMZWg506HJxRnoBE5MhB2SXqtTDog6fLONdoDMxZLrINSGXBlEWhtEUOsw0UWlmVqqwLqLXSEUCueKHVDMGKIg2sfVCsRZZ5o9kls3WIALOAK9Qb0DhTyrqUYZchgzCZIhgF6yVKEBaMVYXxIVcZ0ISlnG0igGdep1QradkMaAaDVzHHVSzLLPNTmC4VkH+r9olLo2PjMtyOvCQCeA5ftnyi2zrnwGD48jzyXLoPc40uvPJmcWfu/DSqH+Bh65tsN+oPzrbZzufOqUG9Emmd6oLvTTrPzMZYzFObVUPQSZrspksphVz4XCbm8OkT0N+gU9F9/vydO3m5PatOAnMmMuQuYfkDPwGXWCzmhRVy4Sl39tMWOkkY/lZ5vibRUZmAgcFUSQ4ugfQqYo221BKCMJgZ3GPilzpvl97dKg6VbntM+v51LqIWp30+pkzRmHmZZZtob+lcBB/6wflQcnbesmR1OU7Z2kz9zb5mNl1O2QKlrQzrN3rjA7rU7qJ909DBa0cXAaxdpC6tLcw4js2w11wQPZaz35Rka9NLVR6Ht+1G1G9EG/K4nJXSljHL+dlpnDGXZKpFdH9Vj+PM5eLiegFUAL2E3ldIUxdRQro7unUBFwbov+ZAviGPE5fOFLW6lOnz+oi67rCI34+M/ckBkHBhkjGQRkdyEqZytFqkZpqqLENvjamQFvWKZWQMZ/2KTOXCTuuwJWNwl2kEcOumax7b/RyRKVjqKEkQ+8h+yJtVAIQLc91695IbVeoUHfNPJCafT7ZzOTJTasyDqp+aCQBSGuTTAnXq5pqJ9mJWFCj5lIscpQN0hox/b3NIA1q0q/rYg2zkgaZpWN1WT7f13iQtynt0bMvw4rDo9+rtbrKQax6uN8C9mnuBv13x1qZWWZYFvSvH2T2SHPPdI/nDTfsVFb0A36xiy7pR/TdEhANz23Bwb4cd3hU0Zlf1im6aCbOvO9QM3bJpwezSxVe8YjrORI0v4ya/mPjm5sb9pw4B9TK1qIMwU4upR7zTKpzJEeTbXhDAbrPim0FvzctdT8ayvJgGkOuYeZDbANyaUqOzUkhJU8427qSS0+j2JDp+DmoRh9pQSGGpKm1R2v8n1e/I8cedwNbv7zpgdBu5ws0gd0ibBC/otG+deEHgVWzzoo2QI4BYFTZma3PsAvOF8JE56qANZTBHNxT4As0B0wHQ1Ffu4/0tgRoHCRoY4EHEVKIxpWlf+xU8cLd+mARW3fgmQyxgFIGroMAMIS3ADMdxvGeiJsXS0cvz5GwwvLgYjkbUDGO2ohX7OPDWpeyZJXhvcvKG0WkYdtf3Mlcczkej4yRNw+AICZBOQoCcDtPRsD/nF7Ozl/PzWcIvztIR47PBbJjMZqPkfHCWji5G8/Nk+HLWn8/Pzkdng3POR7MZ5+f9U/JQT9wSkgbxdv4CVCL0D+Ddy48fP3xswcu6dfF9OPsabe1iwKuf3ry5vLraIapXIm/Dv3+8/pkOzi8GW7ujZLMMp0tkmV16VN+2Pvn79+8+ECCqQIeSwdVc4MvJudIQFoVagGyLYrzB9IHVRYgUAd2YHKpcyTgGc2zq03+dfH5xbNLJGHehHWNtb7daLBaopxnLZ5wd0vivyulrVU0KgdQbsaoLPYTEdJCjZZxZBpTWU1TwFiokrWEyJt3OlkOX3HbhlQO5dSIIUhRskynGwQ26PVNmoU21G3xBNhBypa4RKK319J0JVyJQWjN0yWA2juOGme9FLLwrND0djaZQ0mC9m9vdh9L+hE4nnWe+kHpXbXnpKvADxHW9B4+BdHRK4I7r1yf/j6BMde7I4cN/kP16Ztf/iY+a7dJt+FRMdtYndUDUfbHT0yHQ4MXGV2KtYAgeL4x3+hOngVQWhA2dhNSFplvROBZbMCH3y+QWv6njV/lbFS5p8cBgGfuMJhXf7a40eS4MTlNVbBo38SHhN510ni2RcaBJ//iSpiw+FFXdZtYdRlPBQ6e1y3GztpNFlXV86Vvd+MFCOVA9v1ejPctXzN81Fmect7JUKNDvaHLSmNWgBavGOzqQFv8f9cblAW/WXq9XTd3JmQ+VPiTi/JqLKkasZtK4TQPMMS3X8f3T6UpwVNNUKzn9Vc3qTotvZE+6oRln2Kqi65llNwKwqkyX0AlUTRpqNdO78OrV/vQfVxo/P8R+gXbqcs+Wtb8mdrARmKXDK2QuJD98PEDtpkCYQ8hs3Rc9a7pA81xIoAlQ3yvvwxe4YXphgPYhc2vgCwTfl5AcFs+aWroZMxjSZi3ywQXb4N1RYXxQ7LhT8T+s1SH+sZGsCKCvQVdf4X0IfCX......" -> (known after apply)
# (2 unchanged attributes hidden)
~ part {
# (3 unchanged attributes hidden)
}
}
an example of the aws_instance apply plan
# module.xyz.module.xyz["xyz"].aws_instance.red must be replaced
-/+ resource "aws_instance" "red" {
~ user_data = "f30bd5a600a88aacf234c18f777f8be8b18dd613" -> (known after apply) # forces replacement
There’s nothing immediately obvious that the data block depends on only the terraform local ‘part1’ which is a templatefile.
we’ve tried removing all variables the template file is using and the data block is still being render each apply
@MattTDickinson, the example output shows the data source is nested in some modules – are either of those modules called with depends_on?
Yes actually @jbardin. This child module has a depends_on on another child module
main.tf :
module "configuration_step_function" {
source = "./modules/step_function"
count = var.deploy_step_function == true ? 1 : 0
environment = var.environment
workflow = var.workflow
workflow_no_id = var.workflow_no_id
region = var.region
site = var.site
deploymentid = var.deploymentid
mwcore_master_private_ip = var.mwcore_master_private_ip
mwcore_wrk01_private_ip = var.mwcore_wrk01_private_ip
mwcore_wrk02_private_ip = var.mwcore_wrk02_private_ip
mwedge_license_pool = var.mwedge_license_pool
mwedge_license_account = var.mwedge_license_account
resource_bucket_name = var.resource_bucket_name
resource_tags = var.resource_tags
lambda_security_group_id = var.lambda_security_group_id
subnet_id = var.subnet_id
sub_department = var.sub_department
}
module "non_live_ec2" {
source = "./modules/ec2"
for_each = var.channels
channel = replace(lower(each.key), " ", "-")
subnet_id = var.subnet_id
environment = var.environment
workflow = var.workflow
workflow_no_id = var.workflow_no_id
region = var.region
resource_tags = var.resource_tags
ec2_specific_tags = var.ec2_specific_tags
aws_account_id = var.aws_account_id
resource_bucket_name = var.resource_bucket_name
instance_size = var.instance_sizes.non_live
ami_id = var.ami_id
instance_profile = aws_iam_instance_profile.this.name
route53_zone_id = var.route53_zone_id
site = var.site
sub_department = var.sub_department
security_groups = concat([aws_security_group.this.id], var.additional_security_groups)
alarms_sns_topic_base_name = var.alarms_sns_topic_base_name
additional_resource_tags = merge(
{ service2 = replace(lower(each.key), " ", "-") }, var.additional_resource_tags,
var.deploy_video_monitoring == true ? { "video_testing" : "false" } : {})
s3_endpoint_dns = var.s3_endpoint_dns
depends_on = [module.configuration_step_function]
}
at the end there
Hi @MattTDickinson,
Terraform understands what you’ve configured there as "all actions for module "non_live_ec2" must happen after all actions for module "configuration_step_function".
If there are any planned changes for the step function module then Terraform must wait until the apply phase to do any actions – including reading data sources – needed for the EC2 module.
This configuration seems a little problematic because it will – as a result of all of these smaller interactions – cause the EC2 instance to be replaced every time anything changes about the step function.
The most ideal solution here would be to remove the depends_on entirely and describe the dependencies to Terraform more precisely by passing the necessary values from output values of one module to specific input variables of the other. Terraform can then follow the fine-grain dependency relationships through the individual input variables and output values.
A potentially-less-invasive alternative would be to hoist the data resource up into the root module and pass its result into the second module. The data resource will therefore be outside the scope of the depends_on and can therefore have its own fine-grain dependency relationships despite the second module still using a coarse dependency relationship for the entire module.
@apparentlymart that’s a great explanation thank you. I’ll give that a go and update on here the result.