Terraform thinks the GCP region might have changed... why?

Terraform
1

My overall problem is that terraform wants to destroy and recreate a database because it doesn’t “know” what the region will be until after apply.

I have my main state, and it references two modules.
The main state requires a google provider of “hashicorp/google”. Then it creates a provider resource like such

provider "google" {
  region  = "us-west1"
  project = "worm" 
  zone    = "us-west1-a"

  alias = "worm"
}

It also creates another for a project = “security” with an alias of security.
It then creates this data object

data "google_client_config" "current" {
  provider = google.worm
}

It does a few other small things, like setup an aws provider and use that for a few things.
The big things it does is call the two modules.
For each of those it passes in the providers like so…

module "worm_cluster" {
  source = "modules/gke_cluster"

  cluster_name       = "worm"
  ...

  providers = {
    google                = google.worm
    google.security = google.security
  }
}

and

module "tp_database" {
  source = "modules/gcp_database"

  cluster_name            = module.worm_cluster.cluster_name
  ...

  providers = {
    google      = google.worm
  }

The stuff in the “…” for each of those are either hardcoded values, var.* things, or locals with no dependency on anything in the modules.
Now clearly the database depends on the cluster module since it is pulling values from the cluster module for it’s inputs.

Worm_cluster has this

...
    configuration_aliases = [google, google.security]
...
data "google_client_config" "current" {}

And tp_database has this

#no configuration aliases set
data "google_client_config" "current" {
  provider = google
}

I am assuming the cluster module uses the “google” alias as the provider.
Anyway at various places both modules use

data.google_client_config.current.region

Specifically the tp_database module uses the above region to set the region of a “google_sql_database_instance” resource.
The worm_cluster module of course creates a “google_container_cluster” resource and also uses the region for the location.

Now someone merged in a change that set a property on the “google_container_cluster” to False. That’s it.

Terraform identifies this and says it can do an in-place update on the cluster. All good.
BUT it also wants to completely destroy and recreate the database in tp_database. And the reason is that it says it doesn’t know the region yet.

region                         = "us-west1" -> (known after apply) # forces replacement

That region was set with to “data.google_client_config.current.region”

There are no “depends_on” for either module if that matters.

What is confusing terraform into not being able to “know” the region in one module while it appears to know it just fine in the other?

2

Hi @RandellP,

What is the entire plan output? Or specifically, what exactly does Terraform say for data.google_client_config.current (or whichever one is used to config the region attribute)?

The only reason a data source attribute would be unknown is if it can’t be read during the plan due to pending dependencies, and if there are no references from the data source configuration block, then depends_on is the only other way a dependency could be added to the data source.

3

Other than all the lines about refreshing state… this is the top of the output.

Terraform will perform the following actions:

  # module.tp_database.data.google_client_config.current will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_client_config" "current" {
      + access_token   = (sensitive value)
      + default_labels = (known after apply)
      + id             = (known after apply)
      + project        = (known after apply)
      + region         = (known after apply)
      + zone           = (known after apply)
    }

This next part comes after the above and is heavily redacted because it has a ton of details that aren’t safe to share here…

  # module.tp_database.google_sql_database_instance.service_database_instance must be replaced
-/+ resource "google_sql_database_instance" "service_database_instance" {
      ~ available_maintenance_versions = [
          - "POSTGRES_blah",
        ] -> (known after apply)
      ~ connection_name                = "worm:us-west1:worm-tp-worm" -> (known after apply)
      + dns_name                       = (known after apply)
      + encryption_key_name            = (known after apply)
      ~ first_ip_address               = "123.123.123.123" -> (known after apply)
      ~ id                             = "worm-tp-worm" -> (known after apply)
      ~ instance_type                  = "CLOUD_SQL_INSTANCE" -> (known after apply)
      ~ ip_address                     = [
          - {
              - ip_address     = "123.123.123.123"
              - type           = "PRIVATE"
                # (1 unchanged attribute hidden)
            },
        ] -> (known after apply)
      ~ maintenance_version            = "POSTGRES_blah" -> (known after apply)
      + master_instance_name           = (known after apply)
        name                           = "worm-tp-worm"
      ~ private_ip_address             = "123.123.123.123" -> (known after apply)
      + psc_service_attachment_link    = (known after apply)
      + public_ip_address              = (known after apply)
      ~ region                         = "us-west1" -> (known after apply) # forces replacement
      ~ self_link                      = "https://sqladmin.googleapis.com/sql/v1beta4/projects/worm/instances/worm-tp-worm" -> (known after apply)
      ~ server_ca_cert                 = [
          - {
              - cert             = <<-EOT
                    -----BEGIN CERTIFICATE-----
stuff
                    -----END CERTIFICATE-----
                EOT
              - common_name      = "C=US,O=Google\\, Inc,CN=Google Cloud SQL Server CA,dnQualifier=something"
              - create_time      = "long ago"
              - expiration_time  = "long from now"
              - sha1_fingerprint = "blah"
            },
        ] -> (known after apply)
      ~ service_account_email_address  = "something@gcp-sa-cloud-sql.iam.gserviceaccount.com" -> (known after apply)
        # (4 unchanged attributes hidden)

      ~ replica_configuration (known after apply)

      ~ settings {
          ~ connector_enforcement       = "NOT_REQUIRED" -> (known after apply)
          - deletion_protection_enabled = false -> null
          ~ disk_size                   = meh -> (known after apply)
          ~ version                     = meh -> (known after apply)
            # (11 unchanged attributes hidden)

          ~ backup_configuration {
              - binary_log_enabled             = false -> null
              ~ start_time                     = "soon" -> (known after apply)
              ~ transaction_log_retention_days = 7 -> (known after apply)
                # (3 unchanged attributes hidden)

                # (1 unchanged block hidden)
            }

          ~ ip_configuration {
              - enable_private_path_for_google_cloud_services = false -> null
                # (5 unchanged attributes hidden)
            }

          ~ location_preference (known after apply)
          - location_preference {
              - zone                   = "us-west1-foo" -> null
                # (2 unchanged attributes hidden)
            }

            # (2 unchanged blocks hidden)
        }
    }

  # module.worm_cluster.google_container_cluster.cluster will be updated in-place
  ~ resource "google_container_cluster" "cluster" {
        id                                       = "projects/worm/locations/us-west1/clusters/worm"
        name                                     = "worm"
        # (36 unchanged attributes hidden)

      ~ node_pool_defaults {
          ~ node_config_defaults {
              ~ insecure_kubelet_readonly_port_enabled = "FALSE" -> "TRUE" # THIS IS THE ACTUAL CHANGE THAT TRIGGERS ALL THIS. though originally it was true -> false.
                # (1 unchanged attribute hidden)
            }
        }

        # (20 unchanged blocks hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.
4

ok I found it. But I don’t understand it.

At the top level, the module tp_database has this

depends_on = [google_service_account.temporal_service_account]

And that is defined as

resource "google_service_account" "tp_service_account" {
  account_id   = "${local.tp_deployment_name}-${data.google_client_config.current.region}"
  display_name = "tp ${data.google_client_config.current.region} Service Account"

  provider = google.worm
}

And the tp_database module passes in a attribute of the service account like so

service_account_email   = google_service_account.tp_service_account.email

So the depends on clearly isn’t needed. Commenting it out fixes the issue. But why? Even without it, the module clearly depends on the service account. What is different about the explicit depends_on?

5

The module as a whole does not naturally depend on the google_service_account, only the resources within the module which use the service_account_email value do. When you add depends_on to the module block though, you are declaring that everything within that module depends on all changes to google_service_account.temporal_service_account.

6

I can follow that. sure. But google_service_account.temporal_service_account
wasn’t in the plan output. Why would adding it as a dependency on everything in the module trigger the google_client_config in the module to not know it’s region and such until after apply.

And google_service_account.temporal_service_account mainly depends on the toplevel google_client_config which the module level google_client_config already indirectly depend on because it is derived from the provider passed to the module, which is the same provider used to derive the toplevel google_client_config

7

Unfortunately the snippets here aren’t enough to piece together the full context, so I’m really just guessing as to the exact cause. The only concrete info I have is that the data source is deferred because it: “depends on a resource or a module with changes pending”. That is because it either directly references a managed resource with changes, or there is a depends_on which either directly or transitively includes changes.