One module silently not reading remote state

TF Version - 1.3.7, also tried 1.6.0
Provider AWS

We manage multiple accounts (in AWS but this question appears provider independent), each aws account corresponding to a project, with each aws account managed from a separate terraform directory, by separate tf runs in the project account’s directory.

There’s common information used across all project accounts, which is created and managed as (s3-stored) remote state from another directory, and each project account directory pulls from the remote state for its common context.

It’s worked for years.

But now one account (only) is not seeing refreshed information in the common remote state.

  • The failure to read is silent – the only way I knew there was a problem was that a newly added attribute in remote state was not available in the failing account
  • The failure to read may have started a number of months ago
  • I’ve downloaded and examined the remote state locally, and sure enough, the remote state is quite old
  • Reimporting from a new state would require nearly 100 imports

I inspected the project account state file was puzzled to see the remote state information duplicated in the project account’s directory – I had thought that remote state was dynamically pulled on every plan or apply run, but apparently the project account state keeps a copy or cache of the remote state from the time it was last modified.

I’d like to find a way to either find out why remote state is not being read in this one case, or even better, force it to be reread.

I’d prefer to avoid manual state modification, but will resort to it if required. So far, experiments carried out by migrating the remote state to a local copy of the project account directory show no changes in a plan, even with extreme measures like removing the entire remote state in the statefile. Running plan in such a condition simply produces a normal plan

Ideas?

Hi @rpattcorner,

It’s not entirely clear what the architecture of your configuration is, or what is failing, without seeing the configuration and/or a reproducible example.

I’m assuming by “remote state” here you are referring to the terraform_remote_state data source. When executing Terraform, there is only ever one implementation of that resource regardless of how many modules use it, meaning that if there is any difference in the behavior between modules it is going to be the result of the configuration. It may help to output the configuration values being used to ensure they are what you expect at that location and time.

You also mentioned that you downloaded the remote state and it is “quite old”. If that’s the case there is no way that other references to the same state could return something newer, terraform_remote_state just downloads and parses the state file indicated by the configuration. If other configurations are getting different output values, then they must be pointing at a different remote state.

The remote state you are referring to must be managed by some other Terraform root module, somewhere. That root module which defined the backend storage for that state is where you need to look to see why some outputs may not have been updated to the expected value.