Terraform OpenStack provider: Attaching security groups to instances on first deployment

Hi,

I’m deploying environments in OpenStack using Terraform, and I’ve encountered an issue where the Terraform code doesn’t attach security groups to instances on the first deployment. Following Terraform Apply I can see that the security groups are fully created before the instances are created.

I’m looking for help in finding a solution to this issue. I’ve tried the depends_on parameter in the module that creates the instances to explicitly specify that the instances depend on the creation of the security groups but that didn’t resolve it. I need to run Terraform Apply again to make sure all security rules has been attached to the instances.

Here’s an example of how the Terraform code currently looks:

this is main provider.tf code:

terraform {
  required_version              = ">= 0.14.0"

  backend "http" { }
  required_providers {
    openstack                   = {
      source                    = "terraform-provider-openstack/openstack"
      version                   = "1.47.0"
    }
  }
}

provider "openstack" {
  tenant_name                   = var.tenant_name
  tenant_id                     = var.tenant_id
  user_name                     = var.user_name
  user_domain_name              = var.user_domain_name
  region                        = var.region
  insecure                      = var.allow_insecure_ssl
  auth_url                      = var.auth_url
  application_credential_name   = var.auth_name
  application_credential_secret = var.auth_secret
}

module "network" {
  source                        = "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-network-modules/er-network-tre?ref=main"

  router_id                     = module.network.router_id
  network                       =  {
      name                      = "test_int"
      subnet_name               = "test_subnet"
      cidr                      = "192.345.0.0/24"
  }         
  router                        = {
      name                      = "test_router"
      admin_state_up            = true,
      external_network_id       = "xxxxxxxxxxx"
  }
  dns_ip                        = []
}
module "securityg_default" {
  source                        =  "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-sec-gr-modules/er-securityg-default?ref=v1.1.1"
  }
module "securityg_samba" {
  source                        = "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-sec-gr-modules/er-securityg-samba?ref=v1.1"
  }

module "instance_tre_project" {
	source                        = "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-compute-modules/er-computes-default?ref=v1.2"
  count                         = 2
  name                          = format("%s%02d","er-tre-pp-vm",count.index+1)
	flavor_name                   = "ae7.medium" 
	image_id                      = "1387reree-565186fc"
  key_pair_name                 = ""
	public_ip_network             = "extrnal_4002"

  is_public                     = true
  ports                         = [
    {
			name                      = "${module.network.network_name}",
			network_id                = "${module.network.network_id}",
			subnet_id                 = "${module.network.subnet_id}",
    },
	]

  security_groups               = [
    "${module.securityg_default.security_group_default_name}",
    "${module.securityg_default.security_group_ad_name}",
    "${module.securityg_default.security_group_dns_dhcp_name}",
          ]
  
  depends_on = [
    module.securityg_default,
    module.securityg_samba,
    ]
}

And compute module code

# Create instance
#
resource "openstack_compute_instance_v2" "instance" {
  name            = var.name
  flavor_name     = var.flavor_name
  metadata        = var.metadata
  user_data       = var.user_data

  block_device {
    uuid                  = var.image_id
    source_type           = "image"
    volume_size           = var.block_device_volume_size
    boot_index            = 0
    destination_type      = "volume"
    delete_on_termination = var.block_device_delete_on_termination
  }

  key_pair = var.key_pair_name

  
  dynamic "network" {
    for_each = "${openstack_networking_port_v2.port}"

    content {
      port = network.value["id"]
    }
  }

  security_groups = var.security_groups

}

# Create network port

resource "openstack_networking_port_v2" "port" {
  count = length(var.ports)

  name = var.ports[count.index].name
  network_id = var.ports[count.index].network_id
  admin_state_up = var.ports[count.index].admin_state_up == null ? true : var.ports[count.index].admin_state_up
  #security_group_ids = var.ports[count.index].security_group_ids == null ? [] : var.ports[count.index].security_group_ids
  fixed_ip {
    subnet_id = var.ports[count.index].subnet_id
    ip_address = var.ports[count.index].ip_address == null ? null : var.ports[count.index].ip_address
  }
}

# Create floating ip
resource "openstack_networking_floatingip_v2" "ip" {
  count = var.is_public ? 1 : 0
  pool = var.public_ip_network
}

# Attach floating ip to instance
resource "openstack_compute_floatingip_associate_v2" "ipa" {
  count = var.is_public ? 1: 0
  floating_ip = "${openstack_networking_floatingip_v2.ip[count.index].address}"
  instance_id = "${openstack_compute_instance_v2.instance.id}"
}

Err… you’re setting the input variable of a module to an output variable from the same module? I’m surprised that works!

By the way, it’s redundant to write this as a quoted string interpolation (since you’re just referencing an existing value). It should not be the cause of the problem, but is something you could neaten up.

This should be redundant - Terraform should figure out the requirement for securityg_default since it is referenced in input variables, and you don’t actually appear to be using securityg_samba.


I no longer have access to any OpenStack environments myself to test, but it might prove informative if you were to post the Terraform plan and apply logs for both the initial and second apply that fixes up the security groups.

This can work because by default a module as a whole isn’t a node in the dependency graph, and instead each input variable and output value is a separate node. As long as the module author is careful to keep the dependency chains separate inside the module it is technically possible to have a dependency chain go into a module, out, and then back in again.

In modern Terraform this is a bit more tricky than it used to be because module count, for_each and depends_on do require the whole module to sit behind a single graph node for those arguments to express their dependencies through, and referring to a whole module as a single object in the caller effectively depends on all of its output values and thus creates the effect of the module being a graph node in its own right in the other direction.

So this capability isn’t quite as useful or flexible as it was in Terraform’s early life, but it still works in the simple situations that would have worked in Terraform v0.11 and earlier, as a way to be backward compatible.

Hi Maxb, thanks for looking at it, I have corrected code with your suggestions (see below) but still no luck all its created but it doesn’t actually attach sec groups on first run.

  required_version              = ">= 0.14.0"

  backend "http" { }
  required_providers {
    openstack                   = {
      source                    = "terraform-provider-openstack/openstack"
      version                   = "1.47.0"
    }
  }
}

provider "openstack" {
  tenant_name                   = var.tenant_name
  tenant_id                     = var.tenant_id
  user_name                     = var.user_name
  user_domain_name              = var.user_domain_name
  region                        = var.region
  insecure                      = var.allow_insecure_ssl
  auth_url                      = var.auth_url
  application_credential_name   = var.auth_name
  application_credential_secret = var.auth_secret
}

module "network" {
  source                        = "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-network-modules/er-network-tre?ref=main"

  network                       =  {
      name                      = "er_tre_int"
      subnet_name               = "er_tre_int_subnet"
      cidr                      = "192.0/24"
  }         
  router                        = {
      name                      = "er_test_router"
      admin_state_up            = true,
      external_network_id       = "c43f-08c10"
  }
  dns_ip                        = []
}
module "securityg_default" {
  source                        =  "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-sec-gr-modules/er-securityg-default?ref=v1.1.1"
  }
module "securityg_samba" {
  source                        = "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-sec-gr-modules/er-securityg-samba?ref=v1.1"
  }

module "instance_tre_project" {
	source                        = "git::https://gitlab.ac.uk/infra_terraform/openstack-projects-terraform/tf_modules.git//er-compute-modules/er-computes-default?ref=v1.2"
  count                         = 2
  name                          = format("%s%02d","er-tre-pp-vm",count.index+1)
	flavor_name                   = "ae7.medium" 
	image_id                      = "13876fc"
  key_pair_name                 = ""
	public_ip_network             = "extrnal_4002"

  is_public                     = true
  ports                         = [
    {
			name                      = "${module.network.network_name}",
			network_id                = "${module.network.network_id}",
			subnet_id                 = "${module.network.subnet_id}",
    },
	]

  security_groups               = [
    module.securityg_default.security_group_default_id,
    module.securityg_default.security_group_ad_id,
    module.securityg_default.security_group_dns_dhcp_id,
    module.securityg_samba.security_group_smb_id,

          ]
  }

here is the terraform apply output:

Terraform has been successfully initialized!
module.network.openstack_networking_network_v2.network: Creating...
module.network.openstack_networking_router_v2.router: Creating...
module.network.openstack_networking_network_v2.network: Creation complete after 7s [id=f66944]
module.network.openstack_networking_subnet_v2.subnet: Creating...
module.network.openstack_networking_router_v2.router: Creation complete after 8s [id=009c32]
module.network.openstack_networking_subnet_v2.subnet: Creation complete after 7s [id=88dfcd]
module.network.openstack_networking_router_interface_v2.interface: Creating...
module.network.openstack_networking_router_interface_v2.interface: Creation complete after 8s [id=7de9ee]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_create_ceph: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_rds_nfs: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_data_egress: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_guac: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_smb_guac: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_data_ingress: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad: Creation complete after 1s [id=0d8a]
module.securityg_default.openstack_networking_secgroup_rule_v2.er_sr_out_tcp_addc_10_7: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp: Creation complete after 1s [id=915]
module.securityg_default.openstack_networking_secgroup_rule_v2.er_sr_out_udp_addc_10_7: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default: Creation complete after 1s [id=6bf58]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_rds_nfs: Creation complete after 1s [id=34b1ad2]
module.securityg_default.openstack_networking_secgroup_rule_v2.er_sr_out_udp_addc_10_: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba: Creation complete after 1s [id=b5318f]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_guac: Creation complete after 1s [id=b56da]
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba_create_ceph: Creation complete after 1s [id=08fd]
module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0]: Creating...
module.instance_tre_project[1].openstack_networking_port_v2.port[0]: Creating...
module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0]: Creating...
module.instance_tre_project[0].openstack_networking_port_v2.port[0]: Creating...
module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0]: Creation complete after 6s [id=2823]
module.instance_tre_project[1].openstack_networking_port_v2.port[0]: Creation complete after 7s [id=8e02c883-36bf-431f-af91-c74ebbc80cbf]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Creating...
module.instance_tre_project[0].openstack_networking_port_v2.port[0]: Creation complete after 7s [id=40692aa2-6db3-4ea1-8690-33e6dc3d6d06]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Creating...
module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0]: Creation complete after 7s [id=5baa52b8-8f81-4861-9880-a6300d46d9f6]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Still creating... [10s elapsed]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Still creating... [10s elapsed]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Creation complete after 11s [id=e0910]
module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0]: Creating...
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Creation complete after 12s [id=fdaf4]
module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0]: Creating...
module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0]: Creation complete after 2s [id=10.21110/]
module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0]: Creation complete after 1s [id=10.21236f4/]
Apply complete! Resources: 121 added, 0 changed, 0 destroyed.
Saving cache for successful job
00:04
Creating cache /builds/inreprod-protected...
/builds/infra_terraform/openstack-projects-terraform/tf_tre_tf_preprod/.terraform/: found 564 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Cleaning up project directory and file based variables
00:00
Job succeeded

and output from second terraform apply

module.instance_tre_project[0].openstack_compute_instance_v2.instance: Modifying... [id=8271b297]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Modifying... [id=01d1e63]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Modifications complete after 4s [id=01d8a31e63]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Modifications complete after 5s [id=827bd297]
Apply complete! Resources: 0 added, 2 changed, 0 destroyed.
Saving cache for successful job
00:03
Creating cache /builds/infra_terraform/openstack-projects-terraform/tf_tre_tf_preprod-protected...
/builds/infra_terraform/openstack-projects-terraform/tf_tre_tf_preprod/.terraform/: found 490 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Cleaning up project directory and file based variables
00:01
Job succeeded

I need matching plan logs too to have any reasonable hope of identifying the issue.

please see below:

Plan:

  + create
Terraform will perform the following actions:
  # module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0] will be created
  + resource "openstack_compute_floatingip_associate_v2" "ipa" {
      + floating_ip = (known after apply)
      + id          = (known after apply)
      + instance_id = (known after apply)
      + region      = (known after apply)
    }
  # module.instance_tre_project[0].openstack_compute_instance_v2.instance will be created
  + resource "openstack_compute_instance_v2" "instance" {
      + access_ip_v4        = (known after apply)
      + access_ip_v6        = (known after apply)
      + all_metadata        = (known after apply)
      + all_tags            = (known after apply)
      + availability_zone   = (known after apply)
      + flavor_id           = (known after apply)
      + flavor_name         = "ae7.medium"
      + force_delete        = false
      + id                  = (known after apply)
      + image_id            = (known after apply)
      + image_name          = (known after apply)
      + metadata            = {
          + "RDP Port"     = "3389"
          + "RDP Security" = "test"
        }
      + name                = "er-tre-pp-vm01"
      + power_state         = "active"
      + region              = (known after apply)
      + security_groups     = (known after apply)
      + stop_before_destroy = false
      + user_data           = "da39a3ee"
      + block_device {
          + boot_index            = 0
          + delete_on_termination = false
          + destination_type      = "volume"
          + source_type           = "image"
          + uuid                  = "138148bb"
          + volume_size           = 50
        }
      + network {
          + access_network = false
          + fixed_ip_v4    = (known after apply)
          + fixed_ip_v6    = (known after apply)
          + floating_ip    = (known after apply)
          + mac            = (known after apply)
          + name           = (known after apply)
          + port           = (known after apply)
          + uuid           = (known after apply)
        }
    }
  # module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0] will be created
  + resource "openstack_networking_floatingip_v2" "ip" {
      + address    = (known after apply)
      + all_tags   = (known after apply)
      + dns_domain = (known after apply)
      + dns_name   = (known after apply)
      + fixed_ip   = (known after apply)
      + id         = (known after apply)
      + pool       = "external_4008_tre"
      + port_id    = (known after apply)
      + region     = (known after apply)
      + subnet_id  = (known after apply)
      + tenant_id  = (known after apply)
    }
  # module.instance_tre_project[0].openstack_networking_port_v2.port[0] will be created
  + resource "openstack_networking_port_v2" "port" {
      + admin_state_up         = true
      + all_fixed_ips          = (known after apply)
      + all_security_group_ids = (known after apply)
      + all_tags               = (known after apply)
      + device_id              = (known after apply)
      + device_owner           = (known after apply)
      + dns_assignment         = (known after apply)
      + dns_name               = (known after apply)
      + id                     = (known after apply)
      + mac_address            = (known after apply)
      + name                   = "er_tre_int"
      + network_id             = (known after apply)
      + port_security_enabled  = (known after apply)
      + qos_policy_id          = (known after apply)
      + region                 = (known after apply)
      + tenant_id              = (known after apply)
      + binding {
          + host_id     = (known after apply)
          + profile     = (known after apply)
          + vif_details = (known after apply)
          + vif_type    = (known after apply)
          + vnic_type   = (known after apply)
        }
      + fixed_ip {
          + subnet_id = (known after apply)
        }
    }
  # module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0] will be created
  + resource "openstack_compute_floatingip_associate_v2" "ipa" {
      + floating_ip = (known after apply)
      + id          = (known after apply)
      + instance_id = (known after apply)
      + region      = (known after apply)
    }
  # module.instance_tre_project[1].openstack_compute_instance_v2.instance will be created
  + resource "openstack_compute_instance_v2" "instance" {
      + access_ip_v4        = (known after apply)
      + access_ip_v6        = (known after apply)
      + all_metadata        = (known after apply)
      + all_tags            = (known after apply)
      + availability_zone   = (known after apply)
      + flavor_id           = (known after apply)
      + flavor_name         = "ae7.medium"
      + force_delete        = false
      + id                  = (known after apply)
      + image_id            = (known after apply)
      + image_name          = (known after apply)
      + metadata            = {
          + "RDP Port"     = "3389"
          + "RDP Security" = "test"
        }
      + name                = "er-tre-pp-vm02"
      + power_state         = "active"
      + region              = (known after apply)
      + security_groups     = (known after apply)
      + stop_before_destroy = false
      + user_data           = "da39a3e"
      + block_device {
          + boot_index            = 0
          + delete_on_termination = false
          + destination_type      = "volume"
          + source_type           = "image"
          + uuid                  = "138148bb"
          + volume_size           = 50
        }
      + network {
          + access_network = false
          + fixed_ip_v4    = (known after apply)
          + fixed_ip_v6    = (known after apply)
          + floating_ip    = (known after apply)
          + mac            = (known after apply)
          + name           = (known after apply)
          + port           = (known after apply)
          + uuid           = (known after apply)
        }
    }
  # module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0] will be created
  + resource "openstack_networking_floatingip_v2" "ip" {
      + address    = (known after apply)
      + all_tags   = (known after apply)
      + dns_domain = (known after apply)
      + dns_name   = (known after apply)
      + fixed_ip   = (known after apply)
      + id         = (known after apply)
      + pool       = "external_4008_tre"
      + port_id    = (known after apply)
      + region     = (known after apply)
      + subnet_id  = (known after apply)
      + tenant_id  = (known after apply)
    }
  # module.instance_tre_project[1].openstack_networking_port_v2.port[0] will be created
  + resource "openstack_networking_port_v2" "port" {
      + admin_state_up         = true
      + all_fixed_ips          = (known after apply)
      + all_security_group_ids = (known after apply)
      + all_tags               = (known after apply)
      + device_id              = (known after apply)
      + device_owner           = (known after apply)
      + dns_assignment         = (known after apply)
      + dns_name               = (known after apply)
      + id                     = (known after apply)
      + mac_address            = (known after apply)
      + name                   = "er_tre_int"
      + network_id             = (known after apply)
      + port_security_enabled  = (known after apply)
      + qos_policy_id          = (known after apply)
      + region                 = (known after apply)
      + tenant_id              = (known after apply)
      + binding {
          + host_id     = (known after apply)
          + profile     = (known after apply)
          + vif_details = (known after apply)
          + vif_type    = (known after apply)
          + vnic_type   = (known after apply)
        }
      + fixed_ip {
          + subnet_id = (known after apply)
        }
    }
  # module.network.openstack_networking_network_v2.network will be created
  + resource "openstack_networking_network_v2" "network" {
      + admin_state_up          = (known after apply)
      + all_tags                = (known after apply)
      + availability_zone_hints = (known after apply)
      + dns_domain              = (known after apply)
      + external                = (known after apply)
      + id                      = (known after apply)
      + mtu                     = (known after apply)
      + name                    = "er_tre_int"
      + port_security_enabled   = (known after apply)
      + qos_policy_id           = (known after apply)
      + region                  = (known after apply)
      + shared                  = (known after apply)
      + tenant_id               = (known after apply)
      + transparent_vlan        = (known after apply)
    }
  # module.network.openstack_networking_router_interface_v2.interface will be created
  + resource "openstack_networking_router_interface_v2" "interface" {
      + id        = (known after apply)
      + port_id   = (known after apply)
      + region    = (known after apply)
      + router_id = (known after apply)
      + subnet_id = (known after apply)
    }
  # module.network.openstack_networking_router_v2.router will be created
  + resource "openstack_networking_router_v2" "router" {
      + admin_state_up          = true
      + all_tags                = (known after apply)
      + availability_zone_hints = (known after apply)
      + distributed             = (known after apply)
      + enable_snat             = (known after apply)
      + external_gateway        = (known after apply)
      + external_network_id     = "c476683f"
      + id                      = (known after apply)
      + name                    = "er_tre_router"
      + region                  = (known after apply)
      + tenant_id               = (known after apply)
      + external_fixed_ip {
          + ip_address = (known after apply)
          + subnet_id  = (known after apply)
        }
    }
  # module.network.openstack_networking_subnet_v2.subnet will be created
  + resource "openstack_networking_subnet_v2" "subnet" {
      + all_tags          = (known after apply)
      + cidr              = "192.168.0.0/24"
      + dns_nameservers   = []
      + enable_dhcp       = true
      + gateway_ip        = (known after apply)
      + id                = (known after apply)
      + ip_version        = 4
      + ipv6_address_mode = (known after apply)
      + ipv6_ra_mode      = (known after apply)
      + name              = "er_tre_int_subnet"
      + network_id        = (known after apply)
      + no_gateway        = false
      + region            = (known after apply)
      + tenant_id         = (known after apply)
      + allocation_pool {
          + end   = (known after apply)
          + start = (known after apply)
        }
      + allocation_pools {
          + end   = (known after apply)
          + start = (known after apply)
        }
    }
  
  # module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad will be created
  + resource "openstack_networking_secgroup_v2" "er_tre_sg_ad" {
      + all_tags             = (known after apply)
      + delete_default_rules = true
      + description          = "AD domain controlers in & out security group for tre projects"
      + id                   = (known after apply)
      + name                 = "er_tre_sg_ad"
      + region               = (known after apply)
      + tenant_id            = (known after apply)
    }

  # module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default will be created
  + resource "openstack_networking_secgroup_v2" "er_tre_sg_default" {
      + all_tags             = (known after apply)
      + delete_default_rules = true
      + description          = "default security group for tre projects"
      + id                   = (known after apply)
      + name                 = "er_tre_sg_default"
      + region               = (known after apply)
      + tenant_id            = (known after apply)
    }
  # module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp will be created
  + resource "openstack_networking_secgroup_v2" "er_tre_sg_dns_dhcp" {
      + all_tags             = (known after apply)
      + delete_default_rules = true
      + description          = "test Samba"
      + id                   = (known after apply)
      + name                 = "er_tre_sg_dns_dhcp"
      + region               = (known after apply)
      + tenant_id            = (known after apply)
    }

  # module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba will be created
  + resource "openstack_networking_secgroup_v2" "er_secgroup_samba" {
      + all_tags             = (known after apply)
      + delete_default_rules = true
      + description          = "test 2"
      + id                   = (known after apply)
      + name                 = "er_tre_sec_group_samba"
      + region               = (known after apply)
      + tenant_id            = (known after apply)

Apply

module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0]: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default: Creating...
module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0]: Creating...
module.network.openstack_networking_router_v2.router: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba: Creating...
module.securityg_samba.openstack_networking_secgroup_v2.er_secgroup_samba: Creation complete after 3s [id=82b53654]
module.network.openstack_networking_network_v2.network: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_default: Creation complete after 3s [id=15da1d66]
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad: Creating...
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_dns_dhcp: Creation complete after 4s [id=2facb2b0]
module.securityg_default.openstack_networking_secgroup_v2.er_tre_sg_ad: Creation complete after 1s [id=c1459a12]
module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0]: Creation complete after 9s [id=8f6a03e6]
module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0]: Creation complete after 9s [id=6d971618]
module.network.openstack_networking_network_v2.network: Creation complete after 7s [id=9813cdcc-d392-4d65-8700-d71217eee289]
module.network.openstack_networking_router_v2.router: Still creating... [10s elapsed]
module.network.openstack_networking_router_v2.router: Creation complete after 11s [id=108e857d]
module.network.openstack_networking_subnet_v2.subnet: Creating...
module.network.openstack_networking_subnet_v2.subnet: Creation complete after 6s [id=466ddd8f]
module.instance_tre_project[1].openstack_networking_port_v2.port[0]: Creating...
module.network.openstack_networking_router_interface_v2.interface: Creating...
module.instance_tre_project[0].openstack_networking_port_v2.port[0]: Creating...
module.instance_tre_project[0].openstack_networking_port_v2.port[0]: Creation complete after 6s [id=0d8d2bdf]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Creating...
module.instance_tre_project[1].openstack_networking_port_v2.port[0]: Creation complete after 7s [id=8593463c]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Creating...
module.network.openstack_networking_router_interface_v2.interface: Creation complete after 8s [id=63c264a4]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Still creating... [10s elapsed]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Still creating... [10s elapsed]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Creation complete after 12s [id=827bd394]
module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0]: Creating...
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Creation complete after 12s [id=01d8a315]
module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0]: Creating...
module.instance_tre_project[0].openstack_compute_floatingip_associate_v2.ipa[0]: Creation complete after 2s [827bd394/]
module.instance_tre_project[1].openstack_compute_floatingip_associate_v2.ipa[0]: Creation complete after 3s [01d8a315/]

Apply complete!

second Plan

last "terraform apply" which may have affected this plan:
  
  # module.instance_tre_project[0].openstack_compute_instance_v2.instance has changed
  ~ resource "openstack_compute_instance_v2" "instance" {
        id                  = "827bd394"
        name                = "er-tre-pp-vm01"
      ~ security_groups     = [
          - "15da1d66",
          - "2facb2b0",
          - "c1459a12",
          + "default",
        ]
      + tags                = []
        # (12 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
  # module.instance_tre_project[0].openstack_networking_floatingip_v2.ip[0] has changed
  ~ resource "openstack_networking_floatingip_v2" "ip" {
      + fixed_ip  = "192.168.0.81"
        id        = "6d971618"
      + port_id   = "0d8d2bdf"
      + tags      = []
        # (4 unchanged attributes hidden)
    }
  # module.instance_tre_project[0].openstack_networking_port_v2.port[0] has changed
  ~ resource "openstack_networking_port_v2" "port" {
      + device_id              = "827bd394"
      + device_owner           = "compute:nova"
        id                     = "0d8d2bdf"
        name                   = "er_tre_int"
      + tags                   = []
        # (9 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
  # module.instance_tre_project[1].openstack_compute_instance_v2.instance has changed
  ~ resource "openstack_compute_instance_v2" "instance" {
        id                  = "01d8a315"
        name                = "er-tre-pp-vm02"
      ~ security_groups     = [
          - "15da1d66",
          - "2facb2b0",
          - "c1459a12",
          + "default",
        ]
      + tags                = []
        # (12 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
  # module.instance_tre_project[1].openstack_networking_floatingip_v2.ip[0] has changed
  ~ resource "openstack_networking_floatingip_v2" "ip" {
      + fixed_ip  = "192.168.0.211"
        id        = "8f6a03e6"
      + port_id   = "8593463c"
      + tags      = []
        # (4 unchanged attributes hidden)
    }
  # module.instance_tre_project[1].openstack_networking_port_v2.port[0] has changed
  ~ resource "openstack_networking_port_v2" "port" {
      + device_id              = "01d8a315"
      + device_owner           = "compute:nova"
        id                     = "8593463c"
        name                   = "er_tre_int"
      + tags                   = []
        # (9 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.
─────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
Terraform will perform the following actions:
  
  # module.instance_tre_project[0].openstack_compute_instance_v2.instance will be updated in-place
  ~ resource "openstack_compute_instance_v2" "instance" {
        id                  = "827bd394"
        name                = "er-tre-pp-vm01"
      ~ security_groups     = [
          + "15da1d66",
          + "2facb2b0",
          + "c1459a12",
          - "default",
        ]
        tags                = []
        # (12 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
  # module.instance_tre_project[1].openstack_compute_instance_v2.instance will be updated in-place
  ~ resource "openstack_compute_instance_v2" "instance" {
        id                  = "01d8a315"
        name                = "er-tre-pp-vm02"
      ~ security_groups     = [
          + "15da1d66",
          + "2facb2b0",
          + "c1459a12",
          - "default",
        ]
        tags                = []
        # (12 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
Plan: 0 to add, 2 to change, 0 to destroy.

second Apply

Terraform has been successfully initialized!
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Modifying... [id=827bd394]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Modifying... [id=01d8a315]
module.instance_tre_project[1].openstack_compute_instance_v2.instance: Modifications complete after 4s [id=01d8a315]
module.instance_tre_project[0].openstack_compute_instance_v2.instance: Modifications complete after 5s [id=827bd394]

Apply complete!

From the above logs (the second plan is the really informative one), it can be seen that terraform-provider-openstack truly believed that it had set the security groups up - but then, on the second run, it found the settings had been undone or not taken effect, so it had to do them again.

So, what could cause that?

  • There might be a bug in the provider, causing it to record things in state that aren’t actually true
  • Or, the OpenStack API might have accepted the VM creation, but silently dropped the security group assignment

I do not have the relevant experience to suggest which happened, but hopefully these ideas are of some use to guide further investigations. Perhaps there is some debug logging on the OpenStack side which might give more insight? You could also try running the first apply with Terraform trace logging, as see if there are any hints about how Terraform ends up with a different opinion about the security group assignments to the OpenStack server.

Thank you, Maxb, for your suggestions. I appreciate your help in troubleshooting this issue. It seems that the problem may be related to the introduction of Terraform Modules, and I will investigate this further to try to identify the root cause of the issue. I will also explore some of the other solutions you have suggested to see if they help to resolve the issue. Thank you again for your help and suggestions.

Regards
Michal