bozho
April 21, 2022, 5:48pm
1
Hi all,
When I manage AWS resources with Terraform, I like to implement a “destroy protection” configuration flag, set to true
by default.
The terraform configuration will then set corresponding protection flag on any AWS resources supporting it (e.g. EC2 instances, ELBs, etc.).
This prevents accidental destruction of critical resources by accidentally running terraform destroy
- it requires an operator to first flip the flag in terraform root module, run terraform apply
to disable protections and then run terraform destroy
to destroy infrastructure.
While we do review and approve terraform apply
plans, this approach provides an additional layer of protection for production, while making infra management in dev and test easy (one can simply keep the protection flag set to false
there).
The problem I face now is that I would like to implement the same thing for S3 buckets. I am aware that you cannot delete non-empty buckets, but I’d like to be able to implement explicit protection. Using prevent_destroy
is not an option, as lifecycle
values cannot be variables and our S3 config is implemented as a module.
I know that you can set a bucket policy to prevent deletion, but the problem here is that aws_s3_bucket_policy
is a separate resource, dependent on aws_s3_bucket
, which means that when we run terraform destroy
, the policy resource gets destroyed first, effectively removing deletion protection.
aws_s3_bucket
does have a policy
attribute which would allow me to implement dynamic policy to allow/deny bucket deletion, but the attribute is marked as deprecated in the docs and it seems to be read-only on the current version of the AWS provider.
Is there a way to work around the problem?
Thank you!
Hi @bozho ,
I don’t have a solution but a comment
The aws_s3_bucket
policy attribute was deprecated with AWS provider version 4.x development cycles and basically superseded by the aws_s3_bucket_policy
resource. So your investigation basically raises a downside of those new additional resources.
It might make sense to add your use-case to the discussions on GitHub:
opened 08:31PM - 10 Feb 22 UTC
enhancement
service/s3
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https… ://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
## Update from HashiCorp, 2022-02-25
Hi everyone,
In light of how much traction this issue has gained, and the amount of information contained within, @joe-a-t has graciously allowed us to make edits to the description of this GitHub Issue to better reflect its current state, and so that we may pin it to make it more visible. We would like to sincerely thank him for the opportunity to do so. For the sake of transparency, and to not lose any context, the original issue description and title will be retained beneath this update.
### Breakdown
On 2022-02-10, the AWS Provider team at HashiCorp released a new major version of the AWS Provider for Terraform (`v4.0.0`), which included breaking changes to the `aws_s3_bucket` resource. The update introduced several independent `aws_s3_bucket_*` resources to manipulate the configuration of S3 buckets, where these configuration options previously were arguments to the `aws_s3_bucket` resource itself. This issue spawned out of that change, has received quite a bit of valuable feedback, and is being used as a centralized meeting point for ongoing discussions around these changes.
Information around these changes may be found:
- [Upcoming changes in Version 4.0 of the AWS Provider](https://github.com/hashicorp/terraform-provider-aws/issues/20433)
- [Terraform AWS Provider 4.0 Refactors S3 Bucket Resource (blog post)](https://www.hashicorp.com/blog/terraform-aws-provider-4-0-refactors-s3-bucket-resource)
- [Terraform AWS Provider Version 4 Upgrade Guide](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade)
### Action Items
Given the amount of feedback that we received after the release, the AWS Provider team has been following this issue (as well as any other avenues of feedback we can find) and taking steps to try to alleviate some of the burden placed on operators with this release. The team continues to investigate additional avenues, and will continue to update this issue with additional information where possible. These efforts include:
**Completed**
- [A clarifying comment on what decisions were made, and why they were made](https://github.com/hashicorp/terraform-provider-aws/issues/23106#issuecomment-1036698524)
- An update to the aforementioned blog post to provide more clarification around how resource configuration has changed (namely with regards to configurable -> read-only/computed arguments
- [A clarifying comment on when imports are needed](https://github.com/hashicorp/terraform-provider-aws/issues/23106#issuecomment-1048308419) and an accompanying [update to the migration guide](https://github.com/hashicorp/terraform-provider-aws/pull/23324)
**Currently Investigating**
1) Tooling to help with configuration migration
We are currently investigating potential tooling options to help operators migrate from legacy `aws_s3_bucket` resources to the new `aws_s3_bucket_*` resources. We have identified potential existing tooling and have reached out to the owner of the [tool](https://github.com/minamijoyo/tfedit) to try to help coordinate efforts to make the tool robust enough to share publicly. This is still in the early phases, so the information we can share on it is relatively limited. Despite this, we feel it's appropriate to share that we are looking into it, given the large amount of community interest.
1) Marking deprecated arguments as optional in AWS Provider `v4.x`
This change aims at splitting the difference between the behavior in `v3.x` and `v4.x` by marking deprecated arguments as optional. The goal here is to allow operators more time to transition before the deprecated arguments are fully removed in `v5.x`. Optional arguments would display deprecation warnings, but would not prevent Terraform from running, as was the behavior with the initial `v4.x` releases. Documentation will be written in order to provide as much clarity as possible around how to cope with these changes within configurations.
1) Backporting `aws_s3_bucket_*` resources to AWS Provider `v3.x`
This change aims to bring the new `aws_s3_bucket_*` resources into the `v3.x` series of releases in order to give operators more time to plan and execute migration to the new resources without needing to make the jump to `v4.x`. This would allow operators to use `aws_s3_bucket` resources alongside the new `aws_s3_bucket_*` resources, configuring buckets and updating configurations to the new standards as they see fit, without worry of deprecation notices or failed Terraform operations
Given that backporting resources in this way falls outside of our normal processes, we would like to be explicit around expectations. Bugs in distinct `aws_s3_bucket_*` resources will be backported to ensure functionality, but additional features will not be backported. Documentation will be updated as well to ensure that this expectation is clearly communicated.
## Original issue information
**Title:** Change the S3 bucket refactor to be a new resource instead of modifying the existing one
### Description
Do not change the existing `aws_s3_bucket` resource in v4 and instead provide a new `aws_s3_minimal_bucket` or similar resource that people can use if they want your changes to split out various settings into stand alone resources.
My company has literally thousands of AWS buckets configured through Terraform from hundreds of different root modules. Having to run separate `terraform import` commands for all of the settings you split out into separate resources per your upgrade instructions in https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#s3-bucket-refactor is quite frankly an insane demand to make for users.
Right now, our only options appear to be:
1. Pin to v3 permanently and miss out on all future enhancements/bugfixes/features that the AWS provider publishes.
2. Spend countless hours running (or making our own tool to run) thousands of `terraform import` commands in hundreds of Terraform directories.
Please reconsider this change immediately or at least provide tooling to ease adoption.
### New or Affected Resource(s)
* aws_s3_bucket
### Potential Terraform Configuration
### References
<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
* https://aws.amazon.com/about-aws/whats-new/2018/04/introducing-amazon-ec2-fleet/
--->
* #0000
### Update 2/22/2022
FYI, I (@joe-a-t) chatted with HashiCorp separately and gave them permission to update the issue (including the title and this comment) if they would like the issue to more accurately reflect the work that they are planning on doing.