I’m deploying environments in OpenStack using Terraform, and I’ve encountered an issue where the Terraform code doesn’t attach security groups to instances on the first deployment. Following Terraform Apply I can see that the security groups are fully created before the instances are created.
I’m looking for help in finding a solution to this issue. I’ve tried the depends_on parameter in the module that creates the instances to explicitly specify that the instances depend on the creation of the security groups but that didn’t resolve it. I need to run Terraform Apply again to make sure all security rules has been attached to the instances.
Here’s an example of how the Terraform code currently looks:
Err… you’re setting the input variable of a module to an output variable from the same module? I’m surprised that works!
By the way, it’s redundant to write this as a quoted string interpolation (since you’re just referencing an existing value). It should not be the cause of the problem, but is something you could neaten up.
This should be redundant - Terraform should figure out the requirement for securityg_default since it is referenced in input variables, and you don’t actually appear to be using securityg_samba.
I no longer have access to any OpenStack environments myself to test, but it might prove informative if you were to post the Terraform plan and apply logs for both the initial and second apply that fixes up the security groups.
This can work because by default a module as a whole isn’t a node in the dependency graph, and instead each input variable and output value is a separate node. As long as the module author is careful to keep the dependency chains separate inside the module it is technically possible to have a dependency chain go into a module, out, and then back in again.
In modern Terraform this is a bit more tricky than it used to be because module count, for_each and depends_ondo require the whole module to sit behind a single graph node for those arguments to express their dependencies through, and referring to a whole module as a single object in the caller effectively depends on all of its output values and thus creates the effect of the module being a graph node in its own right in the other direction.
So this capability isn’t quite as useful or flexible as it was in Terraform’s early life, but it still works in the simple situations that would have worked in Terraform v0.11 and earlier, as a way to be backward compatible.
Hi Maxb, thanks for looking at it, I have corrected code with your suggestions (see below) but still no luck all its created but it doesn’t actually attach sec groups on first run.
Terraform has been successfully initialized!
module.network.openstack_networking_network_v2.network: Creating...
module.network.openstack_networking_router_v2.router: Creating...
module.network.openstack_networking_network_v2.network: Creation complete after 7s [id=f66944]
module.network.openstack_networking_subnet_v2.subnet: Creating...
module.network.openstack_networking_router_v2.router: Creation complete after 8s [id=009c32]
module.network.openstack_networking_subnet_v2.subnet: Creation complete after 7s [id=88dfcd]
module.network.openstack_networking_router_interface_v2.interface: Creating...
module.network.openstack_networking_router_interface_v2.interface: Creation complete after 8s [id=7de9ee]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_create_ceph: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_rds_nfs: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_data_egress: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_guac: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_smb_guac: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_data_ingress: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad: Creation complete after 1s [id=0d8a]
module.securityg_default.openstack_networking_secgroup_rule_v2.er_sr_out_tcp_addc_10_7: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp: Creation complete after 1s [id=915]
module.securityg_default.openstack_networking_secgroup_rule_v2.er_sr_out_udp_addc_10_7: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default: Creation complete after 1s [id=6bf58]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_rds_nfs: Creation complete after 1s [id=34b1ad2]
module.securityg_default.openstack_networking_secgroup_rule_v2.er_sr_out_udp_addc_10_: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba: Creation complete after 1s [id=b5318f]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_guac: Creation complete after 1s [id=b56da]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_create_ceph: Creation complete after 1s [id=08fd]
module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0]: Creating...
module.instance_tre_project[1].openstack_networking_port_v2.port[0]: Creating...
module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0]: Creating...
module.instance_tre_project[0].openstack_networking_port_v2.port[0]: Creating...
module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0]: Creation complete after 6s [id=2823]
module.instance_tre_project[1].openstack_networking_port_v2.port[0]: Creation complete after 7s [id=8e02c883-36bf-431f-af91-c74ebbc80cbf]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Creating...
module.instance_tre_project[0].openstack_networking_port_v2.port[0]: Creation complete after 7s [id=40692aa2-6db3-4ea1-8690-33e6dc3d6d06]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Creating...
module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0]: Creation complete after 7s [id=5baa52b8-8f81-4861-9880-a6300d46d9f6]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Still creating... [10s elapsed]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Still creating... [10s elapsed]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Creation complete after 11s [id=e0910]
module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0]: Creating...
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Creation complete after 12s [id=fdaf4]
module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0]: Creating...
module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0]: Creation complete after 2s [id=10.21110/]
module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0]: Creation complete after 1s [id=10.21236f4/]
Apply complete! Resources: 121 added, 0 changed, 0 destroyed.
Saving cache for successful job
00:04
Creating cache /builds/inreprod-protected...
/builds/infra_terraform/openstack-projects-terraform/tf_tre_tf_preprod/.terraform/: found 564 matching artifact files and directories
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Cleaning up project directory and file based variables
00:00
Job succeeded
and output from second terraform apply
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Modifying... [id=8271b297]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Modifying... [id=01d1e63]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Modifications complete after 4s [id=01d8a31e63]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Modifications complete after 5s [id=827bd297]
Apply complete! Resources: 0 added, 2 changed, 0 destroyed.
Saving cache for successful job
00:03
Creating cache /builds/infra_terraform/openstack-projects-terraform/tf_tre_tf_preprod-protected...
/builds/infra_terraform/openstack-projects-terraform/tf_tre_tf_preprod/.terraform/: found 490 matching artifact files and directories
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Cleaning up project directory and file based variables
00:01
Job succeeded
From the above logs (the second plan is the really informative one), it can be seen that terraform-provider-openstack truly believed that it had set the security groups up - but then, on the second run, it found the settings had been undone or not taken effect, so it had to do them again.
So, what could cause that?
There might be a bug in the provider, causing it to record things in state that aren’t actually true
Or, the OpenStack API might have accepted the VM creation, but silently dropped the security group assignment
I do not have the relevant experience to suggest which happened, but hopefully these ideas are of some use to guide further investigations. Perhaps there is some debug logging on the OpenStack side which might give more insight? You could also try running the first apply with Terraform trace logging, as see if there are any hints about how Terraform ends up with a different opinion about the security group assignments to the OpenStack server.
Thank you, Maxb, for your suggestions. I appreciate your help in troubleshooting this issue. It seems that the problem may be related to the introduction of Terraform Modules, and I will investigate this further to try to identify the root cause of the issue. I will also explore some of the other solutions you have suggested to see if they help to resolve the issue. Thank you again for your help and suggestions.