Signature verification with provider network mirrors

I’ve noticed that when you set up a static website as a mirror, the only things present in the mirror files is a hash of the provider, but no signatures.
This had me formulating some questions:

  • Does this mean that there’s no signature verification happening at all when using the provider network mirror protocol?
  • If I want signature verification of provider binaries, do I have any other choice than to set up a self-hosted provider registry implementing the provider registry protocol? (eg. no static file hosts can serve the signature files)

I see that the Plugin Signing docs don’t seem to be clear on this.

Hi @reegnz,

Indeed, if you use a mirror then Terraform assumes that you implicitly trust the mirror and that you will control (by some other means separately from Terraform) which provider packages will be available through the mirror.

The signing process is only for providers published through registries, because the registry itself is part of the chain of trust to the signatures. Terraform verifies that the registry has a valid TLS certificate for the domain in the provider’s source address and therefore is allowed to vouch for signatures for providers in that namespace.

If you intend to publish your own providers with their own signatures then you can run a provider registry on a hostname you control (with a valid TLS certificate for that hostname) and publish signatures via that registry.

If you instead intend to serve third-party providers that originate in some other registry from an alternative source on your local network then you will need to verify the signatures at the time you load the packages into your mirror so that the mirror is guaranteed to only contain correct packages. Terraform itself will only validate the identity of the mirror itself (assuming it uses TLS) and perform an integrity check using the checksums in the mirror’s metadata documents.

The URL scheme of the provider registry protocol does make it slightly harder to implement as a static website, because static web servers typically rely on path suffixes to select MIME types but the provider protocol doesn’t use URL paths ending in .json. However, if you can force your static web server to return Content-type: application/json in some other way then it should be possible to implement that protocol from static files too; it doesn’t require any per-request dynamic behavior.

I was thinking about your response a bit.

So basically I need a mirror if I want to mirror providers from ‘registry.hashicorp.io’, but then I don’t get signature verification, or I set up a private provider registry, but I cannot mirror providers from registry.hashicorp.io with that, only my own providers. That, sadly does not solve my problem.

My primary issue is that implicit trust of a mirror to serve mirrored artifacts is not a good guarantee about the binaries not being tampered with. That’s what the signature of those binaries is for. Hashicorp as the source of the binaries signs them and provides the signature for them as well, and the terraform binary ships with the public keys out-of-band, that can be used when downloading the provider binary. If terraform would verify the provider binary at every run, I wouldn’t have any more questions, but the problem is that it only verifies the binary ONLY if it’s coming from a registry download.

From a pragmatic point-of-view, I do not need HTTP and a trusted mirror, I only need signature verification to ensure I can trust the binary. That’s also how linux distros can still get away with plain HTTP mirrors to this day. Although when you apply a defense in depth mindset, HTTPS does not hurt, but the real value, and proper supply-chain security is all about signatures. Hashicorp builds a binary, signs it, and the end user verifies the binary, under all circumstances.

Maybe that should actually be performed on every run if the terraform config says the provider source is registry.terraform.io, not just on first download? That would also satisfy my paranoid mind. :slight_smile:

Hi @reegnz,

If you always install from the origin registry then Terraform will re-verify the signatures each time. The TLS verification makes sure that only registry.terraform.io can determine what registry.terraform.io/hashicorp/aws represents, but the public key that official providers are signed with is built in to Terraform CLI so the server cannot make CLI report an unofficial provider from any location – even the main registry – appear as “signed by HashiCorp”.

The intention of the mirror model is that “air gapped” production systems exclusively use mirrors and developers working on configuration exclusively use the origin registry, and so the signature verification performed on the developer machine against the registry transfers indirectly to the airgapped system through the checksums in the dependency lock file. The airgapped system can’t directly reach the registry to get the materials necessary to verify a signature, but (unless someone finds a viable SHA256 checksum collision) the checksums indirectly provide the same guarantee.

Some people use different approaches where they exclusively use a mirror even for development, and they perform checksum verification when they build their mirror instead so that the mirror can contain only trusted packages. This centralizes the verification step to whatever team maintains the mirror rather than outsourcing it to individual developer machines. In this model Terraform itself is only verifying checksums for integrity purposes (was the package somehow corrupted), because a configured mirror is always assumed to be trustworthy or else you presumably wouldn’t have configured it. Since there is no authoritative source for what the correct checksums are in this case, folks following this path need to populate their dependency lock files using an explicit extra step like terraform providers lock.

In the end this is all a set of tradeoffs to make it be secure enough by default to let people get started easily but also have some flexibility to handle more complicated situations like airgapped automation environments at the expense of a small amount of explicit configuration.