sodul
May 11, 2022, 11:10pm
1
We use terraform quite extensively and we are currently on Terraform v1.1.9 and use the AWS provider v3.75.1.
We want to upgrade our syntax to be forward compatible with v4 of the aws provider but the sheer amount of terraform import
commands we would have to run is very impractical.
What is the strategy that others have used?
I’m considering to setup a wrapper to automatically run the appropriate import
calls for us.
Some options I have in mind are, after calling init
:
read the existing state file, find deprecated resources and ‘import’ them. That’s a little bit blind as it ignores if the code has been refactored yet, but would apply the import only once.
get the parsable output of plan
to detect removed resources. This is more aware of the refactoring but could be more difficult to parse.
use a combination of both, it is possible that some deprecated resources/attributes will not show in the output of plan
.
I’m working with the upgrade guide, but the fact that we have to manually import/migrate the resources on top of refactoring is problematic. In our case we have dozens of test environments which means that hundreds if not thousands of import
commands would be required, so we need an automated way to address that.
Once we address all syntax warnings and have all the resources imported we will then upgrade the aws provider to v4.
I would imagine that v3.75 of the provider could be extended to automatically detect that the resource was ‘migrated’ and perform the import automatically but that was not done.
How are others handling this migration?
sodul
May 16, 2022, 5:01pm
2
I found this discussion and comment:
opened 08:31PM - 10 Feb 22 UTC
closed 07:57PM - 15 Mar 23 UTC
enhancement
service/s3
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https… ://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
## Update from HashiCorp, 2022-02-25
Hi everyone,
In light of how much traction this issue has gained, and the amount of information contained within, @joe-a-t has graciously allowed us to make edits to the description of this GitHub Issue to better reflect its current state, and so that we may pin it to make it more visible. We would like to sincerely thank him for the opportunity to do so. For the sake of transparency, and to not lose any context, the original issue description and title will be retained beneath this update.
### Breakdown
On 2022-02-10, the AWS Provider team at HashiCorp released a new major version of the AWS Provider for Terraform (`v4.0.0`), which included breaking changes to the `aws_s3_bucket` resource. The update introduced several independent `aws_s3_bucket_*` resources to manipulate the configuration of S3 buckets, where these configuration options previously were arguments to the `aws_s3_bucket` resource itself. This issue spawned out of that change, has received quite a bit of valuable feedback, and is being used as a centralized meeting point for ongoing discussions around these changes.
Information around these changes may be found:
- [Upcoming changes in Version 4.0 of the AWS Provider](https://github.com/hashicorp/terraform-provider-aws/issues/20433)
- [Terraform AWS Provider 4.0 Refactors S3 Bucket Resource (blog post)](https://www.hashicorp.com/blog/terraform-aws-provider-4-0-refactors-s3-bucket-resource)
- [Terraform AWS Provider Version 4 Upgrade Guide](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade)
### Action Items
Given the amount of feedback that we received after the release, the AWS Provider team has been following this issue (as well as any other avenues of feedback we can find) and taking steps to try to alleviate some of the burden placed on operators with this release. The team continues to investigate additional avenues, and will continue to update this issue with additional information where possible. These efforts include:
**Completed**
- [A clarifying comment on what decisions were made, and why they were made](https://github.com/hashicorp/terraform-provider-aws/issues/23106#issuecomment-1036698524)
- An update to the aforementioned blog post to provide more clarification around how resource configuration has changed (namely with regards to configurable -> read-only/computed arguments
- [A clarifying comment on when imports are needed](https://github.com/hashicorp/terraform-provider-aws/issues/23106#issuecomment-1048308419) and an accompanying [update to the migration guide](https://github.com/hashicorp/terraform-provider-aws/pull/23324)
**Currently Investigating**
1) Tooling to help with configuration migration
We are currently investigating potential tooling options to help operators migrate from legacy `aws_s3_bucket` resources to the new `aws_s3_bucket_*` resources. We have identified potential existing tooling and have reached out to the owner of the [tool](https://github.com/minamijoyo/tfedit) to try to help coordinate efforts to make the tool robust enough to share publicly. This is still in the early phases, so the information we can share on it is relatively limited. Despite this, we feel it's appropriate to share that we are looking into it, given the large amount of community interest.
1) Marking deprecated arguments as optional in AWS Provider `v4.x`
This change aims at splitting the difference between the behavior in `v3.x` and `v4.x` by marking deprecated arguments as optional. The goal here is to allow operators more time to transition before the deprecated arguments are fully removed in `v5.x`. Optional arguments would display deprecation warnings, but would not prevent Terraform from running, as was the behavior with the initial `v4.x` releases. Documentation will be written in order to provide as much clarity as possible around how to cope with these changes within configurations.
1) Backporting `aws_s3_bucket_*` resources to AWS Provider `v3.x`
This change aims to bring the new `aws_s3_bucket_*` resources into the `v3.x` series of releases in order to give operators more time to plan and execute migration to the new resources without needing to make the jump to `v4.x`. This would allow operators to use `aws_s3_bucket` resources alongside the new `aws_s3_bucket_*` resources, configuring buckets and updating configurations to the new standards as they see fit, without worry of deprecation notices or failed Terraform operations
Given that backporting resources in this way falls outside of our normal processes, we would like to be explicit around expectations. Bugs in distinct `aws_s3_bucket_*` resources will be backported to ensure functionality, but additional features will not be backported. Documentation will be updated as well to ensure that this expectation is clearly communicated.
## Original issue information
**Title:** Change the S3 bucket refactor to be a new resource instead of modifying the existing one
### Description
Do not change the existing `aws_s3_bucket` resource in v4 and instead provide a new `aws_s3_minimal_bucket` or similar resource that people can use if they want your changes to split out various settings into stand alone resources.
My company has literally thousands of AWS buckets configured through Terraform from hundreds of different root modules. Having to run separate `terraform import` commands for all of the settings you split out into separate resources per your upgrade instructions in https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#s3-bucket-refactor is quite frankly an insane demand to make for users.
Right now, our only options appear to be:
1. Pin to v3 permanently and miss out on all future enhancements/bugfixes/features that the AWS provider publishes.
2. Spend countless hours running (or making our own tool to run) thousands of `terraform import` commands in hundreds of Terraform directories.
Please reconsider this change immediately or at least provide tooling to ease adoption.
### New or Affected Resource(s)
* aws_s3_bucket
### Potential Terraform Configuration
### References
<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
* https://aws.amazon.com/about-aws/whats-new/2018/04/introducing-amazon-ec2-fleet/
--->
* #0000
### Update 2/22/2022
FYI, I (@joe-a-t) chatted with HashiCorp separately and gave them permission to update the issue (including the title and this comment) if they would like the issue to more accurately reflect the work that they are planning on doing.
opened 08:31PM - 10 Feb 22 UTC
closed 07:57PM - 15 Mar 23 UTC
enhancement
service/s3
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https… ://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
## Update from HashiCorp, 2022-02-25
Hi everyone,
In light of how much traction this issue has gained, and the amount of information contained within, @joe-a-t has graciously allowed us to make edits to the description of this GitHub Issue to better reflect its current state, and so that we may pin it to make it more visible. We would like to sincerely thank him for the opportunity to do so. For the sake of transparency, and to not lose any context, the original issue description and title will be retained beneath this update.
### Breakdown
On 2022-02-10, the AWS Provider team at HashiCorp released a new major version of the AWS Provider for Terraform (`v4.0.0`), which included breaking changes to the `aws_s3_bucket` resource. The update introduced several independent `aws_s3_bucket_*` resources to manipulate the configuration of S3 buckets, where these configuration options previously were arguments to the `aws_s3_bucket` resource itself. This issue spawned out of that change, has received quite a bit of valuable feedback, and is being used as a centralized meeting point for ongoing discussions around these changes.
Information around these changes may be found:
- [Upcoming changes in Version 4.0 of the AWS Provider](https://github.com/hashicorp/terraform-provider-aws/issues/20433)
- [Terraform AWS Provider 4.0 Refactors S3 Bucket Resource (blog post)](https://www.hashicorp.com/blog/terraform-aws-provider-4-0-refactors-s3-bucket-resource)
- [Terraform AWS Provider Version 4 Upgrade Guide](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade)
### Action Items
Given the amount of feedback that we received after the release, the AWS Provider team has been following this issue (as well as any other avenues of feedback we can find) and taking steps to try to alleviate some of the burden placed on operators with this release. The team continues to investigate additional avenues, and will continue to update this issue with additional information where possible. These efforts include:
**Completed**
- [A clarifying comment on what decisions were made, and why they were made](https://github.com/hashicorp/terraform-provider-aws/issues/23106#issuecomment-1036698524)
- An update to the aforementioned blog post to provide more clarification around how resource configuration has changed (namely with regards to configurable -> read-only/computed arguments
- [A clarifying comment on when imports are needed](https://github.com/hashicorp/terraform-provider-aws/issues/23106#issuecomment-1048308419) and an accompanying [update to the migration guide](https://github.com/hashicorp/terraform-provider-aws/pull/23324)
**Currently Investigating**
1) Tooling to help with configuration migration
We are currently investigating potential tooling options to help operators migrate from legacy `aws_s3_bucket` resources to the new `aws_s3_bucket_*` resources. We have identified potential existing tooling and have reached out to the owner of the [tool](https://github.com/minamijoyo/tfedit) to try to help coordinate efforts to make the tool robust enough to share publicly. This is still in the early phases, so the information we can share on it is relatively limited. Despite this, we feel it's appropriate to share that we are looking into it, given the large amount of community interest.
1) Marking deprecated arguments as optional in AWS Provider `v4.x`
This change aims at splitting the difference between the behavior in `v3.x` and `v4.x` by marking deprecated arguments as optional. The goal here is to allow operators more time to transition before the deprecated arguments are fully removed in `v5.x`. Optional arguments would display deprecation warnings, but would not prevent Terraform from running, as was the behavior with the initial `v4.x` releases. Documentation will be written in order to provide as much clarity as possible around how to cope with these changes within configurations.
1) Backporting `aws_s3_bucket_*` resources to AWS Provider `v3.x`
This change aims to bring the new `aws_s3_bucket_*` resources into the `v3.x` series of releases in order to give operators more time to plan and execute migration to the new resources without needing to make the jump to `v4.x`. This would allow operators to use `aws_s3_bucket` resources alongside the new `aws_s3_bucket_*` resources, configuring buckets and updating configurations to the new standards as they see fit, without worry of deprecation notices or failed Terraform operations
Given that backporting resources in this way falls outside of our normal processes, we would like to be explicit around expectations. Bugs in distinct `aws_s3_bucket_*` resources will be backported to ensure functionality, but additional features will not be backported. Documentation will be updated as well to ensure that this expectation is clearly communicated.
## Original issue information
**Title:** Change the S3 bucket refactor to be a new resource instead of modifying the existing one
### Description
Do not change the existing `aws_s3_bucket` resource in v4 and instead provide a new `aws_s3_minimal_bucket` or similar resource that people can use if they want your changes to split out various settings into stand alone resources.
My company has literally thousands of AWS buckets configured through Terraform from hundreds of different root modules. Having to run separate `terraform import` commands for all of the settings you split out into separate resources per your upgrade instructions in https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#s3-bucket-refactor is quite frankly an insane demand to make for users.
Right now, our only options appear to be:
1. Pin to v3 permanently and miss out on all future enhancements/bugfixes/features that the AWS provider publishes.
2. Spend countless hours running (or making our own tool to run) thousands of `terraform import` commands in hundreds of Terraform directories.
Please reconsider this change immediately or at least provide tooling to ease adoption.
### New or Affected Resource(s)
* aws_s3_bucket
### Potential Terraform Configuration
### References
<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
* https://aws.amazon.com/about-aws/whats-new/2018/04/introducing-amazon-ec2-fleet/
--->
* #0000
### Update 2/22/2022
FYI, I (@joe-a-t) chatted with HashiCorp separately and gave them permission to update the issue (including the title and this comment) if they would like the issue to more accurately reflect the work that they are planning on doing.
Each of the new aws_s3_bucket_*
resources relies on S3 API calls that utilize a PUT
action in order to modify the target S3 bucket. Because these API calls adhere to standard HTTP methods for REST APIs, they should handle situations where the target configuration already exists (as noted in the HTTP RFC ). Given that this is the case, it’s not strictly necessary to import any new aws_s3_bucket_*
resources that are a one-to-one translation from previous versions of the AWS provider – on the next terraform apply
, they’ll attempt the PUT
, and update the state with the results as necessary.
So it seems that as long as we do a 1 to 1 conversion, everything should work as expected. So in practice the workflow should be:
stop updating your S3 bucket settings
upgrade to the latest v3 provider
run a plan and apply with your latest terraform files
address all the warnings by upgrading to the new s3 resources
run plan and apply again
upgrade to v4
start making changes again