Using a non hashicorp provider in a module

I am trying to use statuscake to setup monitoring of a website. It works fine when I do this in a standalone file. However when I try to break it out into a module, it fails with an error message when I run terraform init. The error message is

Initializing modules...
- statuscake in ..\..\modules\statuscake

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of statuscakedev/statuscake from the dependency lock file
- Finding latest version of hashicorp/statuscake...
- Reusing previous version of hashicorp/azurerm from the dependency lock file
- Installing hashicorp/azurerm v2.49.0...
- Installed hashicorp/azurerm v2.49.0 (signed by HashiCorp)
- Installing statuscakedev/statuscake v1.0.1...
- Installed statuscakedev/statuscake v1.0.1 (self-signed, key ID 4E916C2959C4EAA9)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/statuscake: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/statuscake

If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.

Did you intend to use statuscakedev/statuscake? If so, you must specify that
source address in each module which requires that provider. To see which
modules are currently depending on hashicorp/statuscake, run the following
command:
    terraform providers

The terraform file is

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=2.49.0"
    }
    statuscake = {
      source = "StatusCakeDev/statuscake"
      version = "=1.0.1"
    }
  }
}
provider "azurerm" {
  features {}
} 
provider "statuscake" {
  username = "xxx"
  apikey   = "xxx"
}
module "statuscake" {
  source  = "../../modules/statuscake"

  providers = {
    statuscake = statuscake
  }
}

The module.tf contains this

provider "statuscake" {
  alias = "statuscake_provider"
}
resource "statuscake_test" "test" {
  provider     = statuscake.statuscake_provider
  website_name = "test"
  website_url  = "https://www.google.com"
  test_type    = "HTTP"
  check_rate   = 900
  contact_group  = ["Chris"]
}

Finally the output of terraform modules is as follows. I think that the problem is that the module appears to be trying to reference hashicorp/statuscake, despite the fact that I am passing in a reference to the ‘StatusCakeDev/statuscake’ provider.

Providers required by configuration:
.
├── provider[registry.terraform.io/statuscakedev/statuscake] 1.0.1
├── provider[registry.terraform.io/hashicorp/azurerm] 2.49.0
└── module.statuscake
    └── provider[registry.terraform.io/hashicorp/statuscake]

It’s best practice to have a required_providers block in every module which specifies which providers the module uses. This becomes necessary rather than recommended when the module uses providers from non-HashiCorp namespaces. Adding the required_providers block to your module should fix this issue for you.

More detail: Terraform uses the required_providers block to map your provider local name (statuscake) to the full address (in this case, registry.terraform.io/StatusCakeDev/statuscake). For backwards compatibility reasons, we allow providers to be used without a required_providers entry, but we then need to assume that the host and namespace are registry.terraform.io/hashicorp. As you can see, this assumption is wrong with third party providers.

I hope this makes sense! We’re working right now on improved error messaging for this exact problem, so hopefully it’ll be easier to figure out in the future.

2 Likes

alisdair that worked perfectly. Thanks

A post was split to a new topic: Why doesn’t required_providers in a child module inherit declarations from the parent?

I have the same problem, but I don’t understand the answer. How am I doing this wrong in this example?

terraform {
  required_providers {
    bless = {
      source = "registry.terraform.io/chanzuckerberg/bless"
      version = "~> 0.5"
    }
  }
  required_version = ">= 0.13"
}

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
# // Configure the terraform-provider-bless
provider "bless" {
  region  = data.aws_region.current.name
}

module "bless" {
  // Replace with latest cztack stable release https://github.com/chanzuckerberg/cztack/releases
  source = "github.com/chanzuckerberg/cztack//bless-ca?ref=v0.49.0"

  project = "firehawk-codepipeline"
  service = "bless"
  env     = "dev"
  owner   = "andrewgr"

  region           = data.aws_region.current.name
  authorized_users = ["andrewgr"]
  aws_account_id   = data.aws_caller_identity.current.account_id
}

This produces the following error in terraform .13

│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/bless: provider registry registry.terraform.io does not have a provider
│ named registry.terraform.io/hashicorp/bless
│ 
│ Did you intend to use chanzuckerberg/bless? If so, you must specify that source address in each module which requires that provider. To see
│ which modules are currently depending on hashicorp/bless, run the following command:
│     terraform providers

The terraform/required_providers block is scoped to the individual module.

That means you need to repeat it (but not the provider block) inside each module.

Since cztack/bless-ca at main · chanzuckerberg/cztack · GitHub doesn’t do this, it means that module has not been updated to be compatible with Terraform 0.13+, and you will either need to get the author to update it, or fork it.

I’ve been bitten by this as well, and since no one has explicitly said this, then:

When a child module uses a provider outside of the Hashicorp namespace, then you are required to duplicate the the entire terraform { required_providers { } } code block in every module where that provider is used, regardless if it’s already in the root module’s main.tf file.

In other words, given a file structure of:

|main.tf (root_folder)
|--mymodule_folder
    |--main.tf

Both main.tf files have to each have their own terraform {required_providers{ myprovider = {name,version}}` block, as discussed here, even if that block is an exact duplicate of the block in the root main.tf.

The catch is that this does not apply to any provider within the hashicorp namespace (in other words where the source is hashicorp/). And, what make this very confusing to newcomers to Terraform is that the module tutorial indeed creates an aws resource within a child module that does not re-declare the terraform { required_providers{} } block. So when new users go to create their own module, they get hit with these errors.

In my opinion, this inconsistent behavior should be considered a bug / design flaw. For all the reasons mentioned in the post above, one has to do the same if you come across modules in the hashicorp namespace.

For our company we keep it simple - whether it’s in the hashicorp namespace or not, all modules must have a terraform / required providers block, otherwise it won’t pass testing. I’d recommend getting into the habit of just rewriting it every single time to avoid headaches.

1 Like

It’s a compatibility measure to ease upgrading from earlier Terraform versions.

I’m inclined to agree with you that it’s inelegant that the defaulting behaviour was retained into the 1.x series.

I agree but honestly there was a limited appetite for more breaking changes in the last few releases before 1.0 and so we had to focus on only a few things that would have a significant impact on our ability to keep improving Terraform under the compatibility promises, and this one (along with many others) didn’t make the cut. We had to draw the line somewhere.

I expect we will gradually constrain this over time to make the exception as limited as possible. I already have a draft PR for fixing exactly the current set of providers that existing modules could potentially be using with today’s Terraform so that new providers even in the hashicorp namespace (any created after the first release including that change) would need to be explicitly declared too, but we won’t be able to make this consistent across all providers without a new opt-in language edition, and there are no current plans to do that because the cost doesn’t yet seem to be outweighed by the benefit.

With all of that said, I’d recommend always declaring any providers you are using even if Terraform doesn’t strictly require it today, because it’ll help future readers if your module know which providers you are using without having to know the special backward compatibility rule. This was only intended for existing modules and not for new modules, though I acknowledge that some of the tutorials are outdated themselves and still relying on the backward compatibility behavior; that’s an unfortunate but not surprising consequence of keeping old code working, even when it’s in a tutorial!

I can live with inconsistency, however when there is, the documentation needs to highlight this inconsistency in big red flashing lights. Hours wasted :frowning:

Hi @david7,

Can you share which parts of the documentation you were referring to when you had this problem?

This behavior is mentioned in a few places in the documentation, but of course the problem with implicit behavior is that it’s not something you explicitly chose to rely on and therefore you presumably don’t know where to find the documentation on it. But if you can share what parts of the documentation felt intuitive to you when you were trying to research this then I can share that with the docs team so they can consider whether and how to integrate more information about this into a place where folks are more likely to find it.

Thanks!

I was following the tutorial albeit using a different provider. Build and Use a Local Module | Terraform | HashiCorp Developer.

This is the misleading paragraph :

Notice that there is no provider block in this configuration. When Terraform processes a module block, it will inherit the provider from the enclosing configuration. Because of this, we recommend that you do not include provider blocks in modules.

Thanks @david7!

Indeed, it seems reasonable to me that this tutorial would show the step of creating a versions.tf file that declares the dependency on the hashicorp/aws provider, rather than leaving it implicit.

This also reminds me of one of the technical improvement ideas on our candidate list: adding a new subcommand to allow something like the following as a short, imperative way to generate required_providers entries:

terraform providers add hashicorp/aws

Exact design TBD of course, but the general idea here would be to have this command contact the registry to confirm that the specified provider really exists, to find its current stable version, and then to generate the required_providers and .terraform.lock.hcl entries automatically based on that. This would then make it easier to include this step in tutorials without distracting from the main point of the tutorial by repeating lengthy information that some readers will already know from elsewhere.

I’ll send the tutorial feedback to the docs team and I’ll also open a feature request issue for this terraform providers add subcommand just to have it recorded somewhere more visible than just on an internal ideas list.

Thanks!

Thank you so much! I was trying to figure out what the problem was!!!