I could not find any information on how to create my own source for modules.
I see some are already built in but for example there is no support for Azure Storage where there is support for AWS storage.
Can someone point me on the http url implementation ? and the S3:: implementation?
I need to build my own that will allow me to call specific APIs which requires headers.
I do not see how headers like API Keys can be passed using the HTTP URLs: Module Sources | Terraform by HashiCorp
I like the approach done for S3 bucket where it is using the prefix s3:: but how can I handle my own prefix ?
Hi @fmuntean,
Module installation sources are not a pluggable extension point in Terraform today, and instead we get the various different supported services from an embedded upstream library called go-getter.
The Terraform team doesn’t directly maintain that library, but I expect that if Azure storage support were available in a future version then we would update Terraform to make use of it. As far as I know, the only reason it isn’t supported there yet is because nobody was sufficiently motivated to write the “getter” for it.
With that said, since (as I mentioned) the Terraform team doesn’t directly maintain that library I would suggest opening an issue there if there isn’t one already, to discuss whether there are any blockers I’m not aware of for adding the support.
In the meantime, it may be possible to get what you need using the generic HTTP source, which does essentially the same thing as long as the storage system is HTTP-compatible.
Thanks for getting back to me,
From the look of it the preference is towards the AWS. (everywhere I look in your repos is mostly AWS)
There is a pull request for 2 years now that adds support for Azure blob storage that was never included.
Also for HTTPS these day passing headers is a requirement. Can this feature be added ?
I would propose that this is abstracted maybe a source block that can be referenced instead of the single string.
Then these source blocks can be made pluggable so we can create our own sources as needed, the same like the providers. This way I can create my own HTTPS source where I can send the headers I need in a secure way.
Not everyone wants to be publishing modules to the public.
I already have an API that allows me to return zip files for other purposes and I can reuse the same service to publish/retrieve the TF modules however I have no way to securely do that currently as it requires me to pass certain headers.
Florin
Hi @fmuntean,
I’m sorry that there aren’t more module sources available, and that we have not yet considered designs for making this pluggable.
The bias towards AWS you can see in this case is a matter of timing: Azure storage didn’t exist yet at the time go-getter was originally created. I believe the support for Google Cloud Storage was subsequently contributed at least in part by engineers employed by Google who work on integrations of the GCP features into other software like Terraform.
Since I don’t work on go-getter myself I cannot say why an existing contribution would not have been accepted, but I have since noticed that there is a new GitHub issue in the Terraform repository today with a similar request and I assume you opened that too. I would suggest we continue discussion of the feature there so that it will be visible to the whole Terraform development team.
We have not previously seen a request to make the module installation process extensible with plugins, so there is no existing design work on how that might be achieved either from a language design standpoint or from a plugin protocol standpoint. As you noted, to support it will probably require some new syntax because plugins tend to require more configuration than built-in features do. If you’d like to discuss that I would suggest opening an enhancement request issue in the GitHub repository and we can use that to discuss the design requirements and tradeoffs
Thanks!