How can we avoid existing EBS volumes from being deleted?

Terraform Providers AWS
1

I’m using Terraform 1.1.3 with aws provider 3.75.2 to create TF code for the existing 2-node infra. The code snippet is like below:

module:
resource "aws_ebs_volume" "backend-logs" {
  count = var.create_ebs_log_volumes ? var.backend_nodes_qty : 0

  availability_zone = element(data.aws_subnet.backend.*.availability_zone, count.index)
  size              = var.volume_log_size
  type              = var.ebs_volume_type
  encrypted         = var.ebs_enable_encryption
  kms_key_id        = var.ebs_encryption_key_id
}

resource "aws_volume_attachment" "backend-logs" {
  count       = var.backend_nodes_qty
  device_name = "/dev/sdf"
  volume_id   = element(module.xyz.backend_ebs_volume_log_ids, count.index)
  instance_id = element(module.xyz.backend_instance_ids, count.index)
}

and I’ve imported the instance/volume/attachment resources successfully.

terraform import module.xyz.aws_ebs_volume.backend-logs[0] vol-0123456789abcedf0
terraform import module.xyz.aws_ebs_volume.backend-logs[1] vol-0123456789abcedf1
terraform import aws_volume_attachment.backend-logs[0] /dev/sdf:vol-0123456789abcedf0:i-0123456789abcedf0
terraform import aws_volume_attachment.backend-logs[1] /dev/sdf:vol-0123456789abcedf1:i-0123456789abcedf1

When I run terraform plan, the plan tells me that the volumes are going to be destroyed. How can we avoid that? thanks

  # aws_volume_attachment.backend-logs[0] must be replaced
-/+ resource "aws_volume_attachment" "backend-logs" {
      ~ id          = "vai-1993905001" -> (known after apply)
      ~ volume_id   = "vol-0123456789abcedf0" -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }

  # aws_volume_attachment.backend-logs[1] must be replaced
-/+ resource "aws_volume_attachment" "backend-logs" {
      ~ id          = "vai-1955292002" -> (known after apply)
      ~ volume_id   = "vol-0123456789abcedf1" -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }
    
  # module.xyz.aws_ebs_volume.backend-logs[0] must be replaced
-/+ resource "aws_ebs_volume" "backend-logs" {
      ~ arn                  = "arn:aws:ec2:us-west-2:1234567890:volume/vol-0123456789abcedf0" -> (known after apply)
      ~ availability_zone    = "us-west-2a" -> (known after apply) # forces replacement
      ~ id                   = "vol-0123456789abcedf0" -> (known after apply)
      ~ iops                 = 150 -> (known after apply)
      + kms_key_id           = (known after apply)
      - multi_attach_enabled = false -> null
      + snapshot_id          = (known after apply)
      ~ throughput           = 0 -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.xyz.aws_ebs_volume.backend-logs[1] must be replaced
-/+ resource "aws_ebs_volume" "backend-logs" {
      ~ arn                  = "arn:aws:ec2:us-west-2:1234567890:volume/vol-0123456789abcedf1" -> (known after apply)
      ~ availability_zone    = "us-west-2b" -> (known after apply) # forces replacement
      ~ id                   = "vol-0123456789abcedf1" -> (known after apply)
      ~ iops                 = 150 -> (known after apply)
      + kms_key_id           = (known after apply)
      - multi_attach_enabled = false -> null
      + snapshot_id          = (known after apply)
      ~ throughput           = 0 -> (known after apply)
        # (3 unchanged attributes hidden)
    }
2

What Terraform is telling you, is that it feels a need to make a change to the availability_zone of the volumes, and that is why it wants to recreate them.

I don’t know enough about AWS, or other objects referenced in your setup, to be able to tell if it’s right to do this according to what you have written, or if it’s a bug in the provider’s import support.

I do find it odd that the “forces replacement” attribute is also “known after apply”, not a specific concrete value.

Consider, and please share, what

evaluates to.

3

Ah I think I see the problem - thats evaluating to some nulls, not actual AZ names. Fix that, by specifying the correct AZ names directly, and see how Terraform’s plan changes.

4

I have this data source:

data "aws_subnet" "backend" {
  count = length(aws_instance.backend)
  id    = element(aws_instance.backend.*.subnet_id, count.index)
}
5

Please bear in mind that I have very little experience with AWS… but to me, reading the terraform-provider-aws documentation, it suggests this data source’s availability_zone parameter may be for input filtering only, and may not be usable to query the AZ of a subnet.

6 
resource "aws_instance" "backend" {
  count = var.backend_nodes_qty

  ami           = var.backend_ami
  instance_type = var.backend_instance_type
  monitoring    = var.enable_detailed_monitoring
  key_name      = var.aws_key_pair_name
  subnet_id     = element(var.subnets_private, count.index % length(var.subnets_private))

  iam_instance_profile = var.backend_iam_profile

  vpc_security_group_ids = flatten([
    aws_security_group.backend_server.id,
    var.sg_common_list,
  ])

  root_block_device {
    volume_size           = var.volume_root_size
    volume_type           = var.ebs_volume_type
    encrypted             = var.ebs_enable_encryption
    kms_key_id            = var.ebs_encryption_key_id
    delete_on_termination = false
  }

  lifecycle {
    ignore_changes = [
      volume_tags,
      user_data,
      user_data_base64,
    ]
  }

  user_data_base64 = ""
}
7

Thanks for looking into it!
The code is fine when a new infra is created. The volumes and the instances are generated in the right AZs.

8

Hmm… Perhaps then try comparing the output of terraform state show for an imported volume and a Terraform-created volume.

It may reveal a discrepancy in how the AZ is represented in the state, following import (or at least enable ruling that out)

9

terraform state show is like:

$ terraform state show module.knox-cms.aws_ebs_volume.backend-logs[0]
# module.xyz.aws_ebs_volume.backend-logs[0]:
resource "aws_ebs_volume" "backend-logs" {
    arn                  = "arn:aws:ec2:us-west-2:1234567890:volume/vol-0123456789abcedf0"
    availability_zone    = "us-west-2a"
    encrypted            = false                                                                     
    id                   = "vol-0123456789abcedf0"
    iops                 = 150                 
    multi_attach_enabled = false
    size                 = 50       
    throughput           = 0
    type                 = "gp2"
}

$ terraform state show aws_volume_attachment.backend-logs[0]
# aws_volume_attachment.backend-logs[0]:
resource "aws_volume_attachment" "backend-logs" {
    device_name = "/dev/sdf"
    id          = "vai-1993905001"
    instance_id = "i-0123456789abcedf0"
    volume_id   = "vol-0123456789abcedf0"
}
10

And the comparison with one directly created using Terraform?

11

No, I didn’t create the new ones. I just ran `terraform plan.

  # aws_volume_attachment.backend-logs[0] must be replaced
-/+ resource "aws_volume_attachment" "backend-logs" {
      ~ id          = "vai-1993905001" -> (known after apply)
      ~ volume_id   = "vol-0123456789abcedf0" -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }
12

I would create some new ones and compare the resulting state. It will help to clarify if the issue is a bug in how they were imported or not.

Beyond that, I’m out of ideas on this one.

13

It turned out that ignore_changes has worked around the issue. However it still doesn’t make sense to me because the AZ didn’t change.
What if AZ needs to be changed in other cases?

lifecycle {
    ignore_changes = [ availability_zone ]
}
14

Hi @dnlchen,

I agree that ignore_changes seems more a workaround than a solution here, but I’m also having trouble following all of the indirection through references in your configuration to get a good sense of how the data is flowing.

Can you see in the terraform show output which availability zone each of these objects currently belongs to, and confirm whether they all seem correct? That would at least eliminate the possibility that the imported data was somehow incorrect or inconsistent, and suggest that there is a bug in your configuration instead.

The only other immediate note I have is that further up the discussion you mentioned using data "aws_subnet" "backend" to read in some information about the subnet that each of your instances is connected to. If all you needed from that was the availability zone you might find it helpful to know that aws_instance has its own availability_zone attribute which reports the AZ the instance is running in, so you shouldn’t need to look up that particular value indirectly via the subnet.

15

Hi @apparentlymart

Like I posted above, the AZ of module.knox-cms.aws_ebs_volume.backend-logs[0] didn’t change, and it’s still us-west-2a. module.knox-cms.aws_ebs_volume.backend-logs[1] is in us-west-2b.

Regarding data.aws_subnet, if I don’t use data.aws_subnet.backend, what should I replace element(data.aws_subnet.backend.*.availability_zone, count.index) with?
Is that element(aws_instance.backend.*.availability_zone, count.index)?