During the next fix run hashicorp/aws v5.31.0 infiltrated to the source with some mass code update for upgrade (not just the new provider)
From there we are in the room with
Error: Resource instance managed by newer provider version
The current state of module.wkfingst.aws_cloudwatch_event_rule.this[0] was
created by a newer provider version than is currently selected. Upgrade the
aws provider to work with this state.
Error: Failed to decode resource from state
How it has been fixed:
Try to guess that latest stable state - basically delete top version from the stack;
Got the digest mismatch and upgrade digest in lock table;
Fail with original error;
Repeat steps 1 and 2 till succeed.
Details:
In one case I’ve deleted 2 files, in another 5.
Eventually module got successfully applied on the state that have few records "schema_version": 1 that remains after successful apply.
As you’ve unfortunately learned, the provider protocol’s mechanism for schema upgrades is a “one-way door”: newer provider versions are expected to know how to upgrade the schema from older versions, but older versions cannot anticipate how the schema will evolve later, and so upgrading past a significant provider schema change leads to this situation if you then try to downgrade again.
I’m not personally familiar enough with the details of aws_dynamodb_table to offer specific advice on how to rewrite your state snapshot to return to the previous schema version, but a plausible general approach (with some caveats I’ll discuss after) would be:
Make sure you’re in a working directory that’s using hashicorp/aws v4.28.0. (You can use terraform version after you run terraform init to get a summary of the specific version used for each provider.)
Make sure you know the DynamoDB table name for the resource affected by this problem. Let’s call that TableName for the sake of these instructions.
Run terraform state rm 'module.wkfingst.aws_cloudwatch_event_rule.this[0]' to tell Terraform to “forget about” this object, discarding its entry in the state without deleting the underlying object.
Run terraform import 'module.wkfingst.aws_cloudwatch_event_rule.this[0]' TableName to ask Terraform to adopt the existing object and bind it to the existing resource instance address.
This is the most important step: it will ask the provider (v4.28.0) to retrieve the named table and transform its current state into the form that would be stored in a Terraform state file, in the schema version used by that provider version, and thus what will be written into the state should be compatible with the older version of the provider.
As noted above, there are some caveats to keep in mind:
Not all resource types can support a terraform state rm followed by a terraform import and produce an exactly-equivalent result. This mainly arises for resource types whose underlying APIs have “write-only” attributes, since of course the provider then cannot read them to write them into the state. I don’t think this caveat applies to aws_dynamodb_table, but I’m not 100% sure.
There will briefly be a moment (between the rm and import commands) where your latest state snapshot does not include any object for this resource instance at all. If someone were to run terraform apply during that window, Terraform will propose to create a new DynamoDB table. Therefore if you choose to follow this process you should ensure that nobody else tries to make other changes concurrently with your work.
I would suggest running terraform state pull >backup.tfstate before running terraform state rm just so you’ll have a copy of the latest state snapshot before you change anything. If you find you need to “back out” then you can use terraform state push --force backup.tfstate to restore the previous state snapshot.
This is less important if you are using a state storage backend which retains older versions anyway, but having the state snapshot available locally might still make it easier to recover quickly without having to dig around in e.g. S3 object history.
You directly asked whether a schema_version of 1 is correct for v4.28.0 of the provider. The current schema version for each resource type is typically specified as a hard-coded value inside the relevant provider codebase, and indeed that seems to be true in this case:
So it does appear that a schema_version of 1 is the expected value for the version of the provider you are intending to use.