Disable_api_termination not honored on destroy of AWS EC2 instance

mike.plemmons · June 12, 2023, 8:57pm

Why does terraform destroy tear down an EC2 instance when the disable_api_termination value is set to true? When I set the value I am unable to delete the EC2 instance via the AWS console and the AWS cli. However, when I run terraform destroy the EC2 instance is terminated. I would assume it would fail.

Here is a subset of the output of the terraform state of the instance.

resource "aws_instance" "ec2" {
    ami                                  = "ami-00ee3c71ce62c4e12"
    arn                                  = "arn:aws:ec2:us-east-2:ACCOUNT_ID:instance/i-0228a4e284c4d0629"
    associate_public_ip_address          = true
    availability_zone                    = "us-east-2c"
    cpu_core_count                       = 2
    cpu_threads_per_core                 = 1
    disable_api_stop                     = false
    disable_api_termination              = true   <------------ HERE
    ebs_optimized                        = false
    get_password_data                    = false
    hibernation                          = false
    iam_instance_profile                 = "test-instance-profile"
    id                                   = "i-0228a4e284c4d0629"

When I attempt to delete from the web console I get

Failed to terminate an instance: The instance 'i-0228a4e284c4d0629' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again.

When I attempt to delete from the AWS CLI I get

 ❯ aws ec2 terminate-instances --instance-ids i-0228a4e284c4d0629 --region us-east-2

An error occurred (OperationNotPermitted) when calling the TerminateInstances operation: The instance 'i-0228a4e284c4d0629' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again.

Here is the Terraform destroy output

aws_instance.ec2: Destroying... [id=i-0228a4e284c4d0629]
aws_iam_role_policy_attachment.this_ssm: Destruction complete after 1s
aws_iam_role_policy_attachment.this_s3: Destruction complete after 1s
aws_iam_policy.this: Destroying... [id=arn:aws:iam::847375646079:policy/test-instance-profile]
aws_security_group_rule.egress_one: Destruction complete after 1s
aws_security_group_rule.egress_two: Destruction complete after 1s
aws_iam_policy.this: Destruction complete after 0s
aws_security_group_rule.ingress_one: Destruction complete after 1s
aws_security_group_rule.ingress_two: Destruction complete after 1s
aws_instance.ec2: Still destroying... [id=i-0228a4e284c4d0629, 10s elapsed]
aws_instance.ec2: Still destroying... [id=i-0228a4e284c4d0629, 20s elapsed]
aws_instance.ec2: Still destroying... [id=i-0228a4e284c4d0629, 30s elapsed]
aws_instance.ec2: Still destroying... [id=i-0228a4e284c4d0629, 40s elapsed]
aws_instance.ec2: Still destroying... [id=i-0228a4e284c4d0629, 50s elapsed]
aws_instance.ec2: Destruction complete after 51s
random_shuffle.subnet: Destroying... [id=-]
aws_iam_instance_profile.this: Destroying... [id=test-instance-profile]
aws_security_group.this_two: Destroying... [id=sg-0c94d041efee34325]
aws_security_group.this_one: Destroying... [id=sg-0b44682ed8b69cab6]
random_shuffle.subnet: Destruction complete after 0s
aws_iam_instance_profile.this: Destruction complete after 0s
aws_iam_role.this: Destroying... [id=test-instance-profile]
aws_iam_role.this: Destruction complete after 0s
aws_security_group.this_one: Destruction complete after 0s
aws_security_group.this_two: Destruction complete after 0s

Destroy complete! Resources: 13 destroyed.

apparentlymart · June 12, 2023, 10:51pm

Hi @mike.plemmons,

There seems to be some disagreement among users of the provider as to what they expect this argument to do: some interpret it as you do – “fail destroy unless I first apply an update to disable the production” – and others interpret it as currently implemented, where it’s there to stop tools other than Terraform from terminating the instance.

Some history in these issues:

github.com/hashicorp/terraform-provider-aws

Unable to delete aws_instance if disable_api_termination was true

opened 03:52PM - 07 May 21 UTC

closed 08:25PM - 26 May 22 UTC

tmccombs

bug service/ec2

### Community Note * Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request * Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request * If you are interested in working on this issue or have submitted a pull request, please leave a comment ### Terraform CLI and Terraform AWS Provider Version Terraform: v0.14.8 Provider: v3.38.0 ### Affected Resource(s) * aws_instance ### Terraform Configuration Files Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation. ```hcl resource "aws_instance" "test" { disable_api_termination = true # ... } ``` ### Expected Behavior If I manually enable api termination on instance previously managed by terraform that had `disable_api_termination` set to true, and then apply a change to destroy that instance, the instance should be destroyed, and terraform should not re-enable termination protection as part of the apply. ### Actual Behavior Apply deletion of an aws_instance always modifies the attribute to match what is in the resource config ### Steps to Reproduce 1. `terraform apply` ### Cause I'm pretty sure this was introduced in commit 0a116d70949241082 with this change: ``` - // false = enable api termination - // true = disable api termination (protected) - if !d.Get("disable_api_termination").(bool) { - _, err := conn.ModifyInstanceAttribute(&ec2.ModifyInstanceAttributeInput{ - InstanceId: aws.String(d.Id()), - DisableApiTermination: &ec2.AttributeBooleanValue{ - Value: aws.Bool(d.Get("disable_api_termination").(bool)), - }, - }) + err := resourceAwsInstanceDisableAPITermination(conn, d.Id(), d.Get("disable_api_termination").(bool)) - if err != nil { - log.Printf("[WARN] attempting to terminate EC2 instance (%s) despite error enabling API termination: %s", d.Id(), err) - } + if err != nil { + log.Printf("[WARN] attempting to terminate EC2 instance (%s) despite error enabling API termination: %s", d.Id(), err) } ``` note that it now unconditionally modifies the disable_api_termination attribute, instead of only changing it if disable_api_termination is false. ## Discussion Would it be better to just always set the attribute to false before terminating? Rather than depending on the disable_api_termination setting. (or maybe the intention here was that if disable_api_termination was true in the config, then it should be changed to false when the instance is destroyed?)

github.com/hashicorp/terraform-provider-aws

[Bug]: Please reconsider fix introduced in 4.16.0 for disable_api_termination

opened 09:08AM - 28 Oct 22 UTC

jpbuecken

bug service/ec2

### Terraform Core Version 1.2.4 ### AWS Provider Version 4.21.0 ### Affecte…d Resource(s) * aws_instance ### Expected Behavior If you build your instance with disable_api_termination enabled, it should not be possible to destroy the instance via API (even if it is terraform. The AWS API is used in pipelines or other automatic processes / workflows. E.g. terraform is one tool to use it that way. The idea of disable_api_protection to have an additional layer to protect your instance. For example it protects your instance if you have a a bug in your pipeline code or accidental/wrong usage of workflows. If I try to destroy an instance with disable_api_termination enabled, it should fail. ### Actual Behavior If I try to destroy an instance with disable_api_termination enabled, it succeeds. The fix introduced in 4.16.0 enables api termination unconditionally, making the idea of option disable_api_termination meaningless. Be aware, we use terraform modules, so we cannot use terraform prevent_destroy features as an additional layer. This is a long open issue in terraform (https://github.com/hashicorp/terraform/issues/18367). So please consider if it would have been better to restore the behavior from version 3.37 as described in "Expected Behavior" in https://github.com/hashicorp/terraform-provider-aws/issues/19275 See also https://github.com/hashicorp/terraform-provider-aws/issues/19275#issuecomment-914677748 The problem people faced was that they enabled api termination via AWS console or other tools than terraform, they could not destroy the instance via terraform, because it disabled api termination again, and then tried to destroy the instance. If I build an instance with disable_api_termination = true, then I do it on purpose to protect it for api / command line calls. Since we use terraform only to manage our instances, we wrote workflows that change the value via terraform, then apply, and then destroy the instance, but only in places needed, to avoid accidents. So there is no need to fix it to enable api termination unconditionally. The old behavior allows people to decide if they modify there terraform code to disable api protection (e.g. make it a variable) or use AWS console or other ways before destroy. For me, it looks like https://github.com/hashicorp/terraform-provider-aws/pull/19276#issuecomment-946902292 has only been declined because the code of https://github.com/hashicorp/terraform-provider-aws/pull/19277 was more up to date , but I did not read any consideration if this is a good idea. ### Relevant Error/Panic Output Snippet _No response_ ### Terraform Configuration Files - ### Steps to Reproduce - Build instance with api protection enabled - destroy instance - terraform destroy succeeded without error ### Debug Output _No response_ ### Panic Output _No response_ ### Important Factoids _No response_ ### References _No response_ ### Would you like to implement a fix? _No response_

The first of these requests the current behavior, and the second requests the behavior you wanted.

Given that, I would suggest adding a comment to the second issue to describe your use-case so that the provider development team can consider it. Perhaps there’s a compromise design that could support both interpretations, but deciding that will probably require some additional examples of situations where the current behavior is insufficient.

mike.plemmons · June 13, 2023, 1:22pm

@apparentlymart Thank you for the response. For me, knowing this is the behavior is enough for now but it seems that the documentation ought to have a note that says “Terraform can still destroy the resource when this option is set to True.” I assume if a note like that existed confusion would be avoided.

I was not depending on this functionality before so this did not change how I was working but it seemed unexpected which is why I came here rather than open a bug report first.

mike.plemmons · June 13, 2023, 2:48pm

Interesting finding and not related to EC2 but it appears that there is an inconsistency between different resource deletion protection. On a network load balancer, when I turn on deletion protection on a load balancer, terraform cannot destroy it.

Topic		Replies	Views
AWS Instance is terminated status AWS	2	288	August 27, 2021
Disable_api_termination for Azure Terraform	0	221	June 21, 2022
Issues Destroying EC2 Instances in Terraform Terraform	0	222	July 12, 2020
Terraform AWS ec2 doesn't gracefully shut down Terraform	5	3103	October 9, 2019
Terraform destroying and creating new instance after instance type change AWS	2	895	June 7, 2023

Disable_api_termination not honored on destroy of AWS EC2 instance

Related topics