`aws_db_instance` resource issue: "Cannot find upgrade path" between the same versions

Hi all, we are repeatedly encountering an odd issue when creating RDS instances around snapshots.

We have a module defined with an aws_db_instance resource. That resource has a count of local.num_backup_databases and depends_on an aws_db_snapshot and creates an RDS db around it. When we run apply -target=<this module>, it does the following things:

  • It creates 1 snapshot for each database;
  • It creates 1 RDS instance for each snapshot;

and fails. It produces this odd error message:

│ Error: error modifying DB Instance (backup-es-reindex-dev-db-space-v2-001): InvalidParameterCombination: Cannot find upgrade path from 13.1 to 13.1.
│ 	status code: 400, request id: 227e8303-ad0e-4124-bcee-aa278746c872
│   with module.postgres_space_shards_v2_backup_instances_for_reindex.aws_db_instance.primary_backup[0],
│   on ../modules/postgres/main.tf line 410, in resource "aws_db_instance" "primary_backup":
│  410: resource "aws_db_instance" "primary_backup" {

And marks the created db instances as tainted (the snapshots are fine).

I untaint the resources and run again. This time, it wants to update the created instances with the variables that seem to have been specified in the first place, for instance:

~ monitoring_interval                   = 0 -> 15

Running this second time, it seems to succeed. However, some values are still not set as we’d expect.

The changed values may or may not be related to this; however, we’d very much like to be able to create the instances in one action, at least to eliminate these bizarre 400s as an issue!

We’re currently using TF 1.0.11. We’re generally using the aws provider version 3.64.2; however, I’ve just confirmed that updating to 4.5.0 produces the exact same behaviour.

Does anyone have any thoughts about this odd and somewhat nonsensical error?

1 Like

Is this a net new module for you? We do something similar and out of the blue it started breaking. I was creating new envs and dbs based off our PROD instance which is 12.3. All of a sudden we noticed this starting on March 9:

│ Error: error modifying DB Instance (dev20220309230413854200000001): InvalidParameterCombination: Cannot find upgrade path from 12.3 to 12.3.
│ 	status code: 400, request id: 8948c235-bbc5-40bc-b3bf-6b93cc1bb2cb
│   with module.hr-dev.aws_db_instance.main[0],
│   on modules/xxxxxxxx-stack/database.tf line 159, in resource "aws_db_instance" "main":
│  159: resource "aws_db_instance" "main" {

Yes we are on an older version. But it came out of nowhere - no code changes. All of a sudden a process that worked stopped. It’s caused a huge scramble because I had to go upgrade a bunch of DB versions to fix it.

In DBs there were on 12.5 already we saw the same error.

This feels like a deprecation of some kind on the AWS side. I tried this command:

aws rds describe-db-engine-versions \
  --engine postgres \
  --engine-version 12.3 \
  --query "DBEngineVersions[*].ValidUpgradeTarget[*].{EngineVersion:EngineVersion}" --output text

It returns no results. Until I bump the version to 12.6 I get no results. It feels like this is an internal deprecation of some sort that Terraform is just passing through but isn’t or can’t / won’t do validation on the versions.

aws rds describe-db-engine-versions \
  --engine postgres \
  --engine-version 12.6 \
  --query "DBEngineVersions[*].ValidUpgradeTarget[*].{EngineVersion:EngineVersion}" --output text

In fact, I believe we solved the issue for our case, at least. It looks like a bug in the AWS provider, and it seems to persist in the latest version.

When you create a db from a snapshot, the db will inherit the settings of the snapshot, not just the data. This includes the engine version. If you specify certain values that are inherited, the provider will take note of those and, after it makes the create call, it will make a second ‘modify db’ call with the values that you specify.

The bug is, it doesn’t check to see that the specified engine version is different from the version inherited from the snapshot. So in our case, it saw that we specified 13.1, and thus made an update call after creating the db, attempting to set the settings to 13.1. But it was already at 13.1 (because the db we snapshotted was), and so the update call failed.

I could also imagine this is behavior that AWS silently changed - maybe this kind of call was a no-op in the past.

In any case, we resolved the issue by NOT specifying either engine or engine version in the created db.