I had terraform try to delete a security group that was used by another resource. Outside of terraform, the security group was assigned to a resource. Instead of showing an access denied error, the assumed role token expires, after twenty minutes, and leaves the state unpushed!
Failed to save state: failed to upload state: ExpiredToken: The provided token has expired.
I am using the s3 backend with assume_role.
Full message
aws_security_group.elb: Still destroying... [id=sg-07506a901102e732e, 20m21s elapsed]
Failed to save state: failed to upload state: ExpiredToken: The provided token has expired.
status code: 400, request id: 4E37124AC61B8A0A, host id: RMlQI3jR9JymFUJqo3oybSEn3y6InQnqYEA002aj5UV7r07OhqVH/hlGnWSpdsFP/Eg0JNTJiP8=
Error: Failed to persist state to backend.
The error shown above has prevented Terraform from writing the updated state
to the configured backend. To allow for recovery, the state has been written
to the file "errored.tfstate" in the current working directory.
Running "terraform apply" again at this point will create a forked state,
making it harder to recover.
To retry writing this state, use the following command:
terraform state push errored.tfstate
Releasing state lock. This may take a few moments...
Error releasing the state lock!
Error message: failed to retrieve lock info: ExpiredTokenException: The security token included in the request is expired
status code: 400, request id: 9L9PRP016GHLMUQ2AB7HM8UCP3VV4KQNSO5AEMVJF66Q9ASUAAJG
Terraform acquires a lock when accessing your state to prevent others
running Terraform to potentially modify the state at the same time. An
error occurred while releasing this lock. This could mean that the lock
did or did not release properly. If the lock didn't release properly,
Terraform may not be able to run future commands since it'll appear as if
the lock is held.
In this scenario, please call the "force-unlock" command to unlock the
state manually. This is a very dangerous operation since if it is done
erroneously it could result in two people modifying state at the same time.
Only call this command if you're certain that the unlock above failed and
that no one else is holding a lock.
Here is the provider block.
provider "aws" {
alias = "my_account"
assume_role {role_arn = "arn:aws:iam::1234567890:role/admin/syseng" }
}
The helpful error message was found by a check in CloudTrail for the resource ID.
"errorMessage": "resource sg-07506a901102e732e has a dependent object",
Why can’t terraform tell us the error message, from the provider, and fail gracefully?
What local aws config might I add to prevent this behavior?