Strange resource creation order

Terraform
1

I’m seeing a problem with replacing a resource. The resource is the following EC2 instance:

# module.region_us[0].module.bastion.aws_instance.this[0] must be replaced

Running terraform apply, I see the following steps being done:

Acquiring state lock. This may take a few moments...
module.region_us[0].module.bastion.data.aws_ssm_parameter.this[0]: Reading...
module.region_us[0].module.bastion.data.aws_partition.current: Reading...
module.region_us[0].module.bastion.data.aws_partition.current: Read complete after 0s [id=aws]
module.region_us[0].module.kms_keys["athena"].aws_kms_key.this[0]: Modifying... [id=cebf036c-e0aa-41b5-bec5-b7274a7aef1b]
module.region_us[0].module.kms_keys["secrets"].aws_kms_key.this[0]: Modifying... [id=630d538f-815f-4246-a345-8e229974024d]
module.region_eu[0].module.bastion.data.aws_iam_policy_document.assume_role_policy[0]: Reading...
module.region_us[0].module.bastion.data.aws_iam_policy_document.assume_role_policy[0]: Reading...
module.region_eu[0].module.bastion.data.aws_iam_policy_document.assume_role_policy[0]: Read complete after 0s [id=1256122602]
module.region_us[0].module.bastion.data.aws_iam_policy_document.assume_role_policy[0]: Read complete after 0s [id=1256122602]
module.region_eu[0].module.bastion.data.aws_ssm_parameter.this[0]: Read complete after 0s [id=/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2]
module.region_us[0].module.bastion.data.aws_ssm_parameter.this[0]: Read complete after 1s [id=/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2]
module.region_us[0].module.kms_keys["athena"].aws_kms_key.this[0]: Modifications complete after 6s [id=cebf036c-e0aa-41b5-bec5-b7274a7aef1b]
module.region_us[0].module.kms_keys["secrets"].aws_kms_key.this[0]: Modifications complete after 6s [id=630d538f-815f-4246-a345-8e229974024d]
module.region_us[0].module.bastion.aws_instance.this[0]: Creating...
module.region_us[0].module.glue_table_api_access_logs.aws_glue_catalog_table.this: Modifying... [id=307830480635:loadtest:access_logs_api]
module.region_us[0].module.glue_table_api_access_logs.aws_glue_catalog_table.this: Modifications complete after 1s [id=XXXXXXXXXXXXXX:loadtest:access_logs_api]
module.region_us[0].module.bastion.aws_instance.this[0]: Still creating... [10s elapsed]
module.region_us[0].module.bastion.aws_instance.this[0]: Creation complete after 14s [id=i-0c3c086ea7d1dee9c]
module.region_us[0].aws_volume_attachment.bastion_data_volume: Creating...
module.region_us[0].aws_volume_attachment.bastion_data_volume: Still creating... [10s elapsed]
╷
│ Error: attaching EBS Volume (vol-08ea8d994c78ea9c6) to EC2 Instance (i-0c3c086ea7d1dee9c): VolumeInUse: vol-08ea8d994c78ea9c6 is already attached to an instance
│ status code: 400, request id: 555fe7df-9677-4b7f-8874-2adc80e1fb39
│
│ with module.region_us[0].aws_volume_attachment.bastion_data_volume,
│ on region/bastion.tf line 112, in resource "aws_volume_attachment" "bastion_data_volume":
│ 112: resource "aws_volume_attachment" "bastion_data_volume" {
│

There are no create_before_destroy rules anywhere, but terraform is still trying to create the new EC2 instance and attach the EBS volume to it before destroying the old ones.

The code in question is the following:

module "bastion" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "~> 4.3"

  name = module.names.layers.foundation.bastion_ec2_instance

  create_iam_instance_profile = true
  iam_role_description        = "IAM role for Bastion EC2 instance"
  iam_role_name               = module.names.layers.foundation.bastion_role
  iam_role_policies = {
    AmazonSSMManagedInstanceCore = "arn:${local.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"
    SSMSessionManagerLogs        = aws_iam_policy.session_manager_logs.arn
  }

  ami                    = data.aws_ami.ubuntu.id
  instance_type          = var.bastion_instance_type
  availability_zone      = local.availability_zone
  subnet_id              = element(module.vpc.private_subnets, 0)
  vpc_security_group_ids = [module.bastion_security_group.security_group_id]

  user_data_replace_on_change = true
  user_data_base64            = base64encode(local.bastion_user_data)

  metadata_options = {
    http_endpoint          = "enabled"
    http_tokens            = "required"
    instance_metadata_tags = "enabled"
  }

  monitoring = true

  enable_volume_tags = false
  root_block_device = [
    {
      volume_type = "gp3"
      volume_size = var.bastion_root_volume_size
      encrypted   = true
      kms_key_id  = module.kms_keys["ebs"].key_arn
      tags = {
        Name = module.names.layers.foundation.bastion_root_volume
      }
    },
  ]

  depends_on = [aws_ebs_volume.bastion_data_volume]
}

resource "aws_ebs_volume" "bastion_data_volume" {
  availability_zone = local.availability_zone
  type              = "gp3"
  size              = var.bastion_data_volume_size
  encrypted         = true
  kms_key_id        = module.kms_keys["ebs"].key_arn

  tags = {
    Name = module.names.layers.foundation.bastion_data_volume
  }
}

resource "aws_volume_attachment" "bastion_data_volume" {
  device_name = local.bastion_data_volume_device_name
  volume_id   = aws_ebs_volume.bastion_data_volume.id
  instance_id = module.bastion.id
}

My terraform version is 1.4.6 and the AWS provider is version 4.63.0.

2

When trying to run terraform plan again, I now see this:

# module.region_us[0].module.bastion.aws_instance.this[0] (deposed object 08450a66) will be destroyed

My understanding is that it’s not even possible to get a deposed object without create_before_destroy=true, but as can be seen from the terraform code in my previous post, there’s no lifecycle rule.

I also tried grepping inside the standard terraform EC2 we use and it will not apply a lifecycle rule on the aws_instance resource it creates.

3

Hi @eof,

When Terraform tells you that something “must be replaced” it should also indicate on the next line whether it’s going to destroy first (-/+) or create first (+/-).

Can you see that indication in your plan output? I’d like to first confirm whether Terraform is trying to do create before destroy first, and then we can try to understand why it is doing that.

Thanks!

4

It is correct though that a deposed instance can only be created with create_before_destroy. Your configuration may not have create_before_destroy on any resources, but the module you referenced does. The use of create_before_destroy also requires the re-ordering of upstream resources in the configuration to prevent cycles, so resources which don’t explicitly have the option set may still need to use that ordering.