Terraform custom provider under non-root directory

Terraform
1

I’m looking to use a custom provider within an on-prem network. I can see in the documentation it says to use a layout like:

terraform.example.com/awesomecorp/happycloud

I’m going to want to upload the provider to a shared nexus install where I won’t be able to upload to the root directory, the layout would end up being more like this (for example):

terraform.example.com/repositories/dabesteam/terraform_providers/awesomecorp/happycloud

Is there any way to use a layout more like this without having to do anything like setting up a web server with rewrite rules?

2

Hi @shanedabes,

Are you intending to set up a provider registry for the host terraform.example.com, which would be considered the default direct install location for any providers under terraform.example.com, or are you intending to run a network mirror which serves as an alternative installation source for a provider that would normally be distributed from a third-party registry?

Both approaches allow you to choose whatever base URL you want as the basis for the URLs Terraform will construct, including a base path, but it’s not possible to change the URL patterns Terraform will append after that base path. However, the URL structures for the registry protocol and the network mirror protocol are different, due to the different use-cases, so you might find that one of them is easier to implement than the other within the system you’re working with. (The network mirror protocol was designed to be easy to deploy as a static website from arbitrary files on disk, for example.)

If the software you are using does not allow you to create the URL structure that Terraform is expecting then unfortunately I think an intermediate proxy or redirector would be the only way to integrate with it. There are a few different options with different tradeoffs, if you do end up building a separate adapter service to translate Terraform’s protocol to your backend system:

  • When acting as a client for the provider mirror protocol, Terraform will follow a reasonable number of HTTP redirects when accessing the .json index URLs. Therefore an adapter service could potentially emit redirects to compatible JSON responses under a different URL structure as an alternative to hosting or directly proxying those indexes.
  • If the adapter server is implementing the provider mirror protocol by hosting the JSON indexes itself, you can still include absolute URLs for the provider .zip packages so that they can be retrieved directly from the backend system, rather than from the adapter server.

These details above are assuming you intend to host a provider mirror. The details will be a bit different if you are intending to host an origin registry, but the registry protocol can potentially be deployed in a similar way, albeit with different endpoint URLs and index formats due to the need for an authoritative origin server to provide additional details that a mirror does not.

3

Unless I’m misunderstanding you, I’m looking to set up a provider registry. This would be for custom providers developed in house.

I was under the impression I would just have to match the directory structure specified in the documents, but your link seems to suggest that it’s quite a bit more complicated than that. I was planning on just using a static website (on nexus RM) to host these providers.

I’m not looking to change the URL patterns, I was just hoping I could have a base path after the subdomain.

4

Both of the protocols allow for a base path.

For a provider registry, you can put a URL with a path in the service discovery document for your domain. For example:

{
  "providers.v1": "https://terraform.example.com/repositories/dabesteam/terraform_providers/"
}

If your implementation of the API lives on the same host as the discovery document itself, which seems to be the case in your examples here, then you can use a root-relative URL to specify the location and thus avoid duplicating the hostname:

{
  "providers.v1": "/repositories/dabesteam/terraform_providers/"
}

(Terraform Registry itself uses this root-relative URL approach in the registry.terraform.io discovery document.)

For a provider network mirror, there is no service discovery indirection and the base URL is just specified directly in the CLI configuration’s network_mirror block:

provider_installation {
  network_mirror {
    url = "https://terraform.example.com/repositories/dabesteam/terraform_providers"
  }
}

The provider registry protocol may be harder to implement via static files because it doesn’t use a .json extension for the metadata endpoints and not all static site systems are capable of serving with the application/json content type unless that extension is present.

The network mirror protocol, on the other hand, was intentionally designed to be static-file-hosting friendly by using metadata URLs that end with index.json.

If Nexus’s capabilities are friendly to implementing the registry protocol then I’d recommend it, because that’ll be the easier path for users of the hypothetical terraform.example.com/awesomecorp/happycloud provider. Terraform will, with its default settings, automatically know to talk to the provider registry at terraform.example.com in order to find a package for that provider, and so there’ll be no special CLI configuration required in order to resolve that provider.

However, if you do find that the registry protocol is not implementable withing the capabilities of Nexus you could instead set up a “pseudo-mirror” of providers that belong to a hostname that doesn’t actually have an upstream registry, at the expense of having to explicitly configure Terraform on each machine where you’ll run it so it will use the network mirror for that provider, instead of the direct installation method (which consults the registry host given in the provider address):

provider_installation {
  network_mirror {
    url     = "https://terraform.example.com/repositories/dabesteam/terraform_providers"
    include = ["terraform.example.com/*/*"]
  }
  direct {
    exclude = ["terraform.example.com/*/*"]
  }
}

If you put the above in the CLI configuration (not your module .tf files) then it will tell Terraform to consult the given mirror URL for providers that have the origin host terraform.example.com, but to use the normal direct lookup method for all other providers.

5

A post was split to a new topic: Provider Installation settings in main configuration, not CLI config