Unable to rename multiple aws resources

[MarkKharitonov]

I am following this excellent guide to terraform, specifically I am currently on the 4th part - How to create reusable infrastructure with Terraform modules. And here I have a problem - terraform is unable to rename the resources from the guide. I have no idea whether it is my problem or a bug.

Anyway, here is my directory structure:

C:\Work\terraform> tree /f
Folder PATH listing for volume OSDisk
Volume serial number is 689E-A096
C:.
│   .gitignore
│
├───backend
│       main.tf
│       terraform.tfstate
│
├───modules
│   └───services
│       └───webserver-cluster
│               main.tf
│               outputs.tf
│               variables.tf
│
└───stage
    └───services
        └───webserver-cluster
                main.tf

The backend folder contains the code to setup the S3 backend:

C:\Work\terraform\backend> cat .\main.tf
provider "aws" {
  region = "us-east-2"
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "mark-kharitonov-terraform-up-and-running-state"
  force_destroy = true

  # Enable versioning so we can see the full revision history of our
  # state files
  versioning {
    enabled = true
  }

  # Enable server-side encryption by default
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-up-and-running-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}
C:\Work\terraform\backend> terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.41.0...

...

* provider.aws: version = "~> 2.41"

Terraform has been successfully initialized!

...

C:\Work\terraform\backend> terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_dynamodb_table.terraform_locks will be created
  + resource "aws_dynamodb_table" "terraform_locks" {
...
      + name             = "terraform-up-and-running-locks"
...
    }

  # aws_s3_bucket.terraform_state will be created
  + resource "aws_s3_bucket" "terraform_state" {
...
      + bucket                      = "mark-kharitonov-terraform-up-and-running-state"
...
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_dynamodb_table.terraform_locks: Creating...
aws_s3_bucket.terraform_state: Creating...
aws_dynamodb_table.terraform_locks: Creation complete after 5s [id=terraform-up-and-running-locks]
aws_s3_bucket.terraform_state: Creation complete after 9s [id=mark-kharitonov-terraform-up-and-running-state]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

All is good, the backend is set up.

Next, there is the modules folder. The guide describes its content, but here it is:

C:\work\terraform\modules\services\webserver-cluster> cat .\variables.tf
variable "server_port" {
  description = "The port the server will use for HTTP requests"
  type        = number
  default     = 8080
}

variable "cluster_name" {
  description = "The name to use for all the cluster resources"
  type        = string
}
C:\work\terraform\modules\services\webserver-cluster> cat .\main.tf
data "aws_availability_zones" "all" {}

resource "aws_launch_configuration" "example" {
  image_id        = "ami-0c55b159cbfafe1f0"
  instance_type   = "t2.micro"
  security_groups = [aws_security_group.instance.id]
  user_data       = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p "${var.server_port}" &
              EOF
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "example" {
  launch_configuration = aws_launch_configuration.example.id
  availability_zones   = data.aws_availability_zones.all.names

  min_size = 2
  max_size = 10

  load_balancers    = [aws_elb.example.name]
  health_check_type = "ELB"

  tag {
    key                 = "Name"
    value               = "${var.cluster_name}-asg"
    propagate_at_launch = true
  }
}

resource "aws_elb" "example" {
  name               = "${var.cluster_name}-clb"
  security_groups    = [aws_security_group.elb.id]
  availability_zones = data.aws_availability_zones.all.names

  health_check {
    target              = "HTTP:${var.server_port}/"
    interval            = 30
    timeout             = 3
    healthy_threshold   = 2
    unhealthy_threshold = 2
  }

  # This adds a listener for incoming HTTP requests.
  listener {
    lb_port           = 80
    lb_protocol       = "http"
    instance_port     = var.server_port
    instance_protocol = "http"
  }
}

resource "aws_security_group" "elb" {
  name = "${var.cluster_name}-elb"

  # Allow all outbound
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Inbound HTTP from anywhere
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group" "instance" {
  name = "${var.cluster_name}-instance"
  ingress {
    from_port   = var.server_port
    to_port     = var.server_port
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
C:\work\terraform\modules\services\webserver-cluster> cat .\outputs.tf
output "clb_dns_name" {
  value       = aws_elb.example.dns_name
  description = "The domain name of the load balancer"
}

And now I am using the module to create a cluster in staging named webservers-stage:

C:\work\terraform\stage\services\webserver-cluster> terraform init
Initializing modules...
- webserver_cluster in ..\..\..\modules\services\webserver-cluster

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.41.0...

...

* provider.aws: version = "~> 2.41"

Terraform has been successfully initialized!

...
C:\work\terraform\stage\services\webserver-cluster> terraform apply
module.webserver_cluster.data.aws_availability_zones.all: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.webserver_cluster.aws_autoscaling_group.example will be created
  + resource "aws_autoscaling_group" "example" {
...
      + tag {
          + key                 = "Name"
          + propagate_at_launch = true
          + value               = "webservers-stage-asg"
        }
    }

  # module.webserver_cluster.aws_elb.example will be created
  + resource "aws_elb" "example" {
...
      + name                        = "webservers-stage-clb"
...
    }

  # module.webserver_cluster.aws_launch_configuration.example will be created
  + resource "aws_launch_configuration" "example" {
...
    }

  # module.webserver_cluster.aws_security_group.elb will be created
  + resource "aws_security_group" "elb" {
...
      + name                   = "webservers-stage-elb"
...
    }

  # module.webserver_cluster.aws_security_group.instance will be created
  + resource "aws_security_group" "instance" {
...
      + name                   = "webservers-stage-instance"
...
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.webserver_cluster.aws_security_group.instance: Creating...
module.webserver_cluster.aws_security_group.elb: Creating...
module.webserver_cluster.aws_security_group.instance: Creation complete after 2s [id=sg-0774ace0accdfd348]
module.webserver_cluster.aws_launch_configuration.example: Creating...
module.webserver_cluster.aws_security_group.elb: Creation complete after 2s [id=sg-00e75aa1f2fc5d9e9]
module.webserver_cluster.aws_elb.example: Creating...
module.webserver_cluster.aws_launch_configuration.example: Creation complete after 1s [id=terraform-20191213010022791900000001]
module.webserver_cluster.aws_elb.example: Creation complete after 4s [id=webservers-stage-clb]
module.webserver_cluster.aws_autoscaling_group.example: Creating...
module.webserver_cluster.aws_autoscaling_group.example: Still creating... [10s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Still creating... [20s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Still creating... [30s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Still creating... [40s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Creation complete after 41s [id=tf-asg-20191213010027291700000002]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
Releasing state lock. This may take a few moments...

It’s magic! The cluster is up and running. So far so good.

Now the problematic part. Suppose I want to modify the value of the cluster_name variable effectively renaming all the resources:

C:\work\terraform\stage\services\webserver-cluster> (cat .\main.tf) -replace 'webservers-stage','webservers-stage2' | Out-File -Encoding ascii .\main.tf
C:\work\terraform\stage\services\webserver-cluster> cat .\main.tf
provider "aws" {
  region = "us-east-2"
}

terraform {
  backend "s3" {
    # Replace this with your bucket name!
    bucket = "mark-kharitonov-terraform-up-and-running-state"
    key    = "stage/services/webserver-cluster/terraform.tfstate"
    region = "us-east-2"
    # Replace this with your DynamoDB table name!
    dynamodb_table = "terraform-up-and-running-locks"
    encrypt        = true
  }
}

module "webserver_cluster" {
  source       = "../../../modules/services/webserver-cluster"
  cluster_name = "webservers-stage2"
}

Now the desire is for the cluster name to be webservers-stage2. Here is what happens:

C:\work\terraform\stage\services\webserver-cluster> terraform.exe apply
module.webserver_cluster.data.aws_availability_zones.all: Refreshing state...
module.webserver_cluster.aws_security_group.elb: Refreshing state... [id=sg-00e75aa1f2fc5d9e9]
module.webserver_cluster.aws_security_group.instance: Refreshing state... [id=sg-0774ace0accdfd348]
module.webserver_cluster.aws_elb.example: Refreshing state... [id=webservers-stage-clb]
module.webserver_cluster.aws_launch_configuration.example: Refreshing state... [id=terraform-20191213010022791900000001]
module.webserver_cluster.aws_autoscaling_group.example: Refreshing state... [id=tf-asg-20191213010027291700000002]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.webserver_cluster.aws_autoscaling_group.example will be updated in-place
  ~ resource "aws_autoscaling_group" "example" {
        arn                       = "arn:aws:autoscaling:us-east-2:170091157278:autoScalingGroup:5fe93e07-27d2-4d4e-91cc-24ae916ec735:autoScalingGroupName/tf-asg-20191213010027291700000002"
        availability_zones        = [
            "us-east-2a",
            "us-east-2b",
            "us-east-2c",
        ]
        default_cooldown          = 300
        desired_capacity          = 2
        enabled_metrics           = []
        force_delete              = false
        health_check_grace_period = 300
        health_check_type         = "ELB"
        id                        = "tf-asg-20191213010027291700000002"
      ~ launch_configuration      = "terraform-20191213010022791900000001" -> (known after apply)
      ~ load_balancers            = [
          - "webservers-stage-clb",
          + "webservers-stage2-clb",
        ]
        max_instance_lifetime     = 0
        max_size                  = 10
        metrics_granularity       = "1Minute"
        min_size                  = 2
        name                      = "tf-asg-20191213010027291700000002"
        protect_from_scale_in     = false
        service_linked_role_arn   = "arn:aws:iam::170091157278:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        suspended_processes       = []
        target_group_arns         = []
        termination_policies      = []
        vpc_zone_identifier       = []
        wait_for_capacity_timeout = "10m"

      - tag {
          - key                 = "Name" -> null
          - propagate_at_launch = true -> null
          - value               = "webservers-stage-asg" -> null
        }
      + tag {
          + key                 = "Name"
          + propagate_at_launch = true
          + value               = "webservers-stage2-asg"
        }
    }

  # module.webserver_cluster.aws_elb.example must be replaced
-/+ resource "aws_elb" "example" {
      ~ arn                         = "arn:aws:elasticloadbalancing:us-east-2:170091157278:loadbalancer/webservers-stage-clb" -> (known after apply)
        availability_zones          = [
            "us-east-2a",
            "us-east-2b",
            "us-east-2c",
        ]
        connection_draining         = false
        connection_draining_timeout = 300
        cross_zone_load_balancing   = true
      ~ dns_name                    = "webservers-stage-clb-641972551.us-east-2.elb.amazonaws.com" -> (known after apply)
      ~ id                          = "webservers-stage-clb" -> (known after apply)
        idle_timeout                = 60
      ~ instances                   = [
          - "i-00617fcc06a5ae64f",
          - "i-0ee5eb03f3bf733b0",
        ] -> (known after apply)
      ~ internal                    = false -> (known after apply)
      ~ name                        = "webservers-stage-clb" -> "webservers-stage2-clb" # forces replacement
      ~ security_groups             = [
          - "sg-00e75aa1f2fc5d9e9",
        ] -> (known after apply)
      ~ source_security_group       = "170091157278/webservers-stage-elb" -> (known after apply)
      ~ source_security_group_id    = "sg-00e75aa1f2fc5d9e9" -> (known after apply)
      ~ subnets                     = [
          - "subnet-3e7c2244",
          - "subnet-6569ef29",
          - "subnet-74e8081f",
        ] -> (known after apply)
      - tags                        = {} -> null
      ~ zone_id                     = "Z3AADJGX6KTTL2" -> (known after apply)

        health_check {
            healthy_threshold   = 2
            interval            = 30
            target              = "HTTP:8080/"
            timeout             = 3
            unhealthy_threshold = 2
        }

        listener {
            instance_port     = 8080
            instance_protocol = "http"
            lb_port           = 80
            lb_protocol       = "http"
        }
    }

  # module.webserver_cluster.aws_launch_configuration.example must be replaced
+/- resource "aws_launch_configuration" "example" {
        associate_public_ip_address      = false
      ~ ebs_optimized                    = false -> (known after apply)
        enable_monitoring                = true
      ~ id                               = "terraform-20191213010022791900000001" -> (known after apply)
        image_id                         = "ami-0c55b159cbfafe1f0"
        instance_type                    = "t2.micro"
      + key_name                         = (known after apply)
      ~ name                             = "terraform-20191213010022791900000001" -> (known after apply)
      ~ security_groups                  = [
          - "sg-0774ace0accdfd348",
        ] -> (known after apply) # forces replacement
        user_data                        = "398ce7cb244926b5b22c0dcb00d885ac509c0ee5"
      - vpc_classic_link_security_groups = [] -> null

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + root_block_device {
          + delete_on_termination = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }
    }

  # module.webserver_cluster.aws_security_group.elb must be replaced
-/+ resource "aws_security_group" "elb" {
      ~ arn                    = "arn:aws:ec2:us-east-2:170091157278:security-group/sg-00e75aa1f2fc5d9e9" -> (known after apply)
        description            = "Managed by Terraform"
        egress                 = [
            {
                cidr_blocks      = [
                    "0.0.0.0/0",
                ]
                description      = ""
                from_port        = 0
                ipv6_cidr_blocks = []
                prefix_list_ids  = []
                protocol         = "-1"
                security_groups  = []
                self             = false
                to_port          = 0
            },
        ]
      ~ id                     = "sg-00e75aa1f2fc5d9e9" -> (known after apply)
        ingress                = [
            {
                cidr_blocks      = [
                    "0.0.0.0/0",
                ]
                description      = ""
                from_port        = 80
                ipv6_cidr_blocks = []
                prefix_list_ids  = []
                protocol         = "tcp"
                security_groups  = []
                self             = false
                to_port          = 80
            },
        ]
      ~ name                   = "webservers-stage-elb" -> "webservers-stage2-elb" # forces replacement
      ~ owner_id               = "170091157278" -> (known after apply)
        revoke_rules_on_delete = false
      - tags                   = {} -> null
      ~ vpc_id                 = "vpc-e8a15983" -> (known after apply)
    }

  # module.webserver_cluster.aws_security_group.instance must be replaced
+/- resource "aws_security_group" "instance" {
      ~ arn                    = "arn:aws:ec2:us-east-2:170091157278:security-group/sg-0774ace0accdfd348" -> (known after apply)
        description            = "Managed by Terraform"
      ~ egress                 = [] -> (known after apply)
      ~ id                     = "sg-0774ace0accdfd348" -> (known after apply)
        ingress                = [
            {
                cidr_blocks      = [
                    "0.0.0.0/0",
                ]
                description      = ""
                from_port        = 8080
                ipv6_cidr_blocks = []
                prefix_list_ids  = []
                protocol         = "tcp"
                security_groups  = []
                self             = false
                to_port          = 8080
            },
        ]
      ~ name                   = "webservers-stage-instance" -> "webservers-stage2-instance" # forces replacement
      ~ owner_id               = "170091157278" -> (known after apply)
        revoke_rules_on_delete = false
      - tags                   = {} -> null
      ~ vpc_id                 = "vpc-e8a15983" -> (known after apply)
    }

Plan: 4 to add, 1 to change, 4 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.webserver_cluster.aws_elb.example: Destroying... [id=webservers-stage-clb]
module.webserver_cluster.aws_security_group.instance: Creating...
module.webserver_cluster.aws_elb.example: Destruction complete after 1s
module.webserver_cluster.aws_security_group.elb: Destroying... [id=sg-00e75aa1f2fc5d9e9]
module.webserver_cluster.aws_security_group.instance: Creation complete after 2s [id=sg-0b3b7c4ceb54ac416]
module.webserver_cluster.aws_launch_configuration.example: Creating...
module.webserver_cluster.aws_launch_configuration.example: Creation complete after 1s [id=terraform-20191213015144456000000001]
module.webserver_cluster.aws_security_group.elb: Still destroying... [id=sg-00e75aa1f2fc5d9e9, 10s elapsed]
module.webserver_cluster.aws_security_group.elb: Still destroying... [id=sg-00e75aa1f2fc5d9e9, 20s elapsed]
module.webserver_cluster.aws_security_group.elb: Destruction complete after 28s
module.webserver_cluster.aws_security_group.elb: Creating...
module.webserver_cluster.aws_security_group.elb: Creation complete after 2s [id=sg-06f4f489b60ba9134]
module.webserver_cluster.aws_elb.example: Creating...
module.webserver_cluster.aws_elb.example: Creation complete after 4s [id=webservers-stage2-clb]
module.webserver_cluster.aws_autoscaling_group.example: Modifying... [id=tf-asg-20191213010027291700000002]
module.webserver_cluster.aws_autoscaling_group.example: Still modifying... [id=tf-asg-20191213010027291700000002, 10s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Still modifying... [id=tf-asg-20191213010027291700000002, 20s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Still modifying... [id=tf-asg-20191213010027291700000002, 30s elapsed]
module.webserver_cluster.aws_autoscaling_group.example: Modifications complete after 39s [id=tf-asg-20191213010027291700000002]
module.webserver_cluster.aws_launch_configuration.example: Destroying... [id=terraform-20191213010022791900000001]
module.webserver_cluster.aws_launch_configuration.example: Destruction complete after 0s
module.webserver_cluster.aws_security_group.instance: Destroying... [id=sg-0774ace0accdfd348]
module.webserver_cluster.aws_security_group.instance: Still destroying... [id=sg-0774ace0accdfd348, 10s elapsed]
...
module.webserver_cluster.aws_security_group.instance: Still destroying... [id=sg-0774ace0accdfd348, 10m0s elapsed]

Error: Error deleting security group: DependencyViolation: resource sg-0774ace0accdfd348 has a dependent object
        status code: 400, request id: d8801db8-65c1-4de1-9f7a-107b2cad247f


Releasing state lock. This may take a few moments...

What am I missing?

I opened a bug in Terraform - https://github.com/hashicorp/terraform/issues/23678, but apparently this is not the proper way. I also have this post on SO - https://stackoverflow.com/questions/59315467/terraform-unable-to-rename-multiple-aws-resources

I have the trace log available. I can encrypt it with the same HashiCorp public key and upload to my gist.

Thank you.

[bentterp]

I think I see it, the security group can’t be destroyed because there are still server instances using it.
Try to take the cluster down to 0 members first.

Yes, I know that causes a service disruption, but the Gruntworks tutorial is probably not intended for day-two operations in real production.

[MarkKharitonov]

Please, accept my apologies if the question is stupid, but what is the proper process here? I will parameterize the max_size and min_size properties but then what? Do I:

1. Invoke apply while setting them to 0
2. Wait for successful application
3. Then rename the cluster, restore the sizes and reapply?

Or is there a better way? It does not seem very scalable. Is it possible to correct the code so that rename does this logic automatically? It seems strange that aws provider does not know about this dependency.

[bentterp]

If I were to do something like this in real production, I’d probably make the ELB a separate configuration, and have the servers and aws_lb_listening_rule in another configuration.

That way it would be relatively smooth to stand up a new cluster and move it into service, kinda like a Blue-Green deployment.

Of course, the easiest way of changing the name using the code of the published tutorial would be to do a destroy first, then change the name and do a new apply.

[MarkKharitonov]

Doing destroy first would certainly work, but I still do not understand why there is a need for this in the first place. I thought the provider is supposed to know the dependencies and take care of the resource upgrade correctly. I.e. take care of the details like in-place vs destroy/create and the order of the changes.

I will try the separation.

[MarkKharitonov]

I cannot do the separation right now. I have just started learning terraform and I use AWS only because the tutorial does, so I do not know much about the non classic load balancer that you mention indirectly through aws_lb_listening_rule resource.

Would it be too much to ask you if this is possible with the classic load balancer used in the tutorial? If you could convert the tutorial example from using the classic lb to using the modern lb that would have been absolutely fantastic.