Terraform "Catch-22": Lack of default provider causes error; default provider breaks per-region providers

Terraform
1

(Cross-posted to Stack Overflow for better answer-ranking.)

I have a Catch-22 situation with Terraform:

  1. If I don’t define a default AWS provider (that I don’t actually need) I get errors about the default provider not being defined
  2. If I then define a default AWS provider, it breaks the use of the default AWS provider in modules, that have been overridden with aliased region-specific providers as they are passed in to the module

Versions

The problems happen with both of these Terraform versions:

Terraform v1.7.1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.31.0

Terraform v1.6.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.31.0

Scenarios

(All instances of <AWS_REGION> are actually replaced with a real AWS region identifier, such as us-east-1 or otherwise.)

Scenario #1: No default AWS provider specified:

There are several per-AWS-region aliased AWS providers defined in the root module

provider "aws" {
  alias = "<AWS_REGION>"
  region = "<AWS_REGION>"
}

# Note: There is no default AWS provider, only aliased ones

There are also several per-AWS-region modules with an overridden AWS provider passed in:

# ./modules.tf
module "provider_<AWS_REGION>" {
  source    = "./<AWS_REGION>"

  # AWS per-region provider is passed in to the module, replacing its default provider
  providers = {
    aws = aws.<AWS_REGION>
  }
}

Each module has all of the per-AWS-region configuration, here is an example:

# Should use the default AWS provider `aws`, which has been replaced 

# ./<AWS_REGION>/vpc.tf
resource "aws_vpc" "name_of_vpc" {
  # (configuration)
}

The local Terraform state has module and provider specified:

# ./terraform.tfstate
{
  "version": 4,
  "terraform_version": "1.7.0",
  "serial": 1459,
  "lineage": "<SOME_GUID>",
  "outputs": {},
  "resources": [
    {
      "module": "module.provider_<AWS_REGION>",
      "mode": "managed",
      "type": "<TERRAFORM_RESOURCE_TYPE>",
      "name": "<TERRAFORM_RESOURCE_NAME>",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      # ...
}

When running terraform plan (after a clean init), the following errors are emitted:

Error: When no default provider is specified:

(When this doesn’t exist: provider "aws" { region = "<REGION>" })

│ Error: Invalid provider configuration
│
│ Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and
│ configure the provider's required arguments as described in the provider documentation.

╷
│ Error: Invalid AWS Region:
│
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on <empty> line 0:
│   (source code not available)

Scenario #2: Default provider is added:

This default provider is added:

./provider.tf
provider "aws" {
  region = "<ARBITRARY_AWS_REGION>"
}

Result:

There are no actual errors, but each region-specific module just thinks its AWS provider is for the arbitrary AWS region of the default AWS provider, when it seems like it should actually be whatever AWS-region-specific provider has been passed in to the module.

Scenario #3: Adding .<AWS_REGION> to the end of the provider value in terraform.tfstate:

This error is emitted:

│                                                                                                                              │ To work with
│ module.provider_<AWS_REGION>.<TERRAFORM_RESOURCE_TYPE>.<TERRAFORM_RESOURCE_NAME>                                                                                                                   │ its original provider configuration at module.provider_us-east-1.provider["registry.terraform.io/hashicorp/aws"].<AWS_REGION>
│ is required, but it has been removed. This occurs when a provider configuration is removed while objects created by that     
│ provider still exist in the state. Re-add the provider configuration to destroy                                             
│ module.provider_us-east-1.<AWS_REGION>.<TERRAFORM_RESOURCE_TYPE>.<TERRAFORM_RESOURCE_NAME>,                                                                                                                  │ after which you can remove the provider configuration again.

Scenario #4: Manually specifying provider = aws.<AWS_REGION> to the body of every resource in the AWS-region-specific modules

Doesn’t help: It emits the Error: Provider configuration not present error.

Scenario #5: Combining Scenario #3 and #4

Doesn’t help: It emits the Error: Provider configuration not present error.

2

Hi @nhooey,

The fact that Terraform seems to be trying to configure an implicit default provider instance for hashicorp/aws – implicit in the sense that you have no provider block for it – suggests that something in your configuration is associated with that default provider configuration.

A typical cause of that is forgetting to add an explicit provider argument to one of your resources, or providers argument to one of your module calls. I don’t have other ideas off the top of my head, but if you are sure that explanation doesn’t apply to your configuration then we could try to find another.

If you’ve set providers in a module block where the left side of the provider assignment is just aws (the child module’s default configuration for that provider) then indeed you don’t need to write explicit provider arguments in the child module resources; the child module has its own independent “view” of the provider configurations where a default configuration is present.

Manually editing the terraform.tfstate file will make no difference, because the configuration is authoritative for which resources belong to which provider configurations. The record of that in the state is only for when the whole resource block has been removed and so the configuration no longer says anything about which provider should be used to destroy that existing object.

Running the terraform providers command might give some clues as to what is depending on what in your configuration.

4

Thanks @apparentlymart for the advice.

Scenario #4 actually solved the problem: explicitly specifying a per-AWS-region provider for every resource in every per-AWS-region module made the providers work for the first time. I was just thrown off by all of the planned destruction events that started showing up.

I do think there’s an opportunity to improve the details in a lot of the error messages to make this easier.

This all comes from trying to import all of the AWS infrastructure in to Terraform using Terracognita, since Terracognita doesn’t support importing from multiple regions at the same time. It’s been further complicated by the fact that Elastic Beanstalk automatically creates and destroys resources like aws_autoscaling_group, and I need to figure out a way to tell Terraform to ignore all of those things.

Do you know how to get Terraform to ignore stuff created by Elastic Beanstalk, so Terraform doesn’t want to destroy them, when planning after they have changed automatically?

5

I asked about ignoring entire resources in this thread: