Hi ,
can some one provide better example for how to use inhouse provider , referred this https://www.terraform.io/upgrade-guides/0-13.html#in-house-providers not helpful.
for example i want to use source like below
terraform {
required_providers {
xyz = {
source = “a.b.c.com/repository/artifacts/ix/xyz”
}
}
}
Getting this error:
The “source” attribute must be in the format “[hostname/][namespace/]name”
Thanks
Hi @babuamudala,
As the message says, the source argument must be in a particular form that identifies the hostname, namespace, and provider type. The value you set here isn’t correct syntax, and instead seems to be just a URL with the scheme removed and that doesn’t match what Terraform requires.
If you are trying to follow the guidance of putting the in-house provider in your local filesystem instead of publishing it in a registry then you can assign it any namespace you like under a domain you control and then place the plugin package at the filesystem location described in the upgrade guide.
For example, if you decide to call your provider a.b.c.com/ix/xyz – where ix is the namespace and xyz is the type – then the expected layout of the local filesystem directory for plugins would be:
a.b.c.com/ix/xyz/1.0.0/linux_amd64/terraform-provider-xyz_v1.0.0
For the above I’m really just repeating information from the upgrade guide and so I expect it’s not sufficient to answer your question either, but hopefully if you can try the above and then tell me what problem you encountered trying it I will be able to offer some more specific advice based on what error messages you see.
@apparentlymart
Thank you for responding
I tried as you suggested
terraform {
required_providers {
xyaws = {
source = “nexus.xyz.com/artifacts/xyaws”
version = “1.0.0”
}
infoblox = {
source = “nexus.xyz.com/artifacts/inflobox”
version = “3.0.0”
}
}
}
Local file system directory is like this
nexus.xyz.com/artifacts/xyaws/1.0.0/linux_amd64/terraform-provider-xyaws
Getting like this , fyi tried other ways too. same error
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
nexus.xyz.com/artifacts/inflobox: could not connect to nexus.nml.com:
“https://nexus.nml.com/.well-known/terraform.json”: x509: certificate signed
could you tell what i am missing here
Thanks,
shobhan.
Hi @babuamudala!
The error message you saw suggests that Terraform correctly understood your intent to use the provider nexus.xyz.com/artifacts/inflobox, but that for some reason it wasn’t able to detect that the provider is available locally and so it tried to treat nexus.xyz.com (or nexus.nml.com? I assume this inconsistency is just a typo in your comment) as a Terraform provider registry.
The question then is why Terraform isn’t detecting the provider as installed locally. One way to debug that would be to set the environment variable TF_LOG=trace before you run terraform init again. The trace log output is verbose, but an early part of that output is from Terraform scanning the local filesystem directories to see which providers are available there. On my system it looks like this, because I haven’t actually created any of the local search directories or placed providers into them:
2020/10/14 09:35:40 [INFO] Terraform version: 0.14.0 dev
2020/10/14 09:35:40 [INFO] Go runtime version: go1.15.2
2020/10/14 09:35:40 [INFO] CLI args: []string{"/home/mart/go/1.15.2/bin/terraform", "init"}
2020/10/14 09:35:40 [DEBUG] Attempting to open CLI config file: /home/mart/.terraformrc
2020/10/14 09:35:40 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory /home/username/.terraform.d/plugins
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory /home/username/.local/share/terraform/plugins
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory /usr/share/gnome-classic/terraform/plugins
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2020/10/14 09:35:40 [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins
2020/10/14 09:35:40 [INFO] CLI command args: []string{"init"}
Because I’m on an Ubuntu Linux system there are some additional search directories for me that come from how Ubuntu implements the “XDG Base Directory” specification, but that detail aside the idea here is that Terraform is searching one or more directories to see what providers are installed in them, and then any provider it finds in that scanning will be excluded from Terraform’s attempts to install providers from registries over the network.
If you’re not sure how to interpret the log output, could you share the initial part of the log that appears before the line [INFO] CLI command args: []string{"init"}? (the parts after that seem less important for the problem we’re currently working on).
yes it is just typo in comments.
i did traced as you requested , here is the output after running
2020/10/14 20:49:33 [INFO] Terraform version: 0.13.4
[INFO] Go runtime version: go1.14.7
[INFO] CLI args: []string{"/var/terraform13/terraform13", "validate"}
[DEBUG] Attempting to open CLI config file: /root/.terraformrc
[DEBUG] File doesn't exist, but doesn't need to. Ignoring.
[DEBUG] checking for credentials in "/root/.terraform.d/plugins"
[DEBUG] ignoring non-existing provider search directory terraform.d/plugins
[DEBUG] will search for provider plugins in /root/.terraform.d/plugins
[TRACE] getproviders.SearchLocalDirectory: /root/.terraform.d/plugins is a symlink to /root/.terraform.d/plugins
[DEBUG] ignoring non-existing provider search directory /root/.local/share/terraform/plugins
[DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
[DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
Thanks,
shobhan
Tried this way too
Local file system moved here /root/.terraform.d/plugins
Now local filesystem directory looks like this
/root/.terraform.d/plugins/nexus.xyz.com/artifacts/infloblox/2.0.0/linux_amd64/terraform-provider-infoblox_v2.0.0.
requirement syntax
terraform {
required_version = ">= 0.13.0"
required_providers {
infoblox = {
# source is required for providers in other namespaces, to avoid ambiguity.
source = "nexus.xyz.com/artifacts/inflobox"
version = "2.0.0"
}
}
}
Getting same error.
Could not retrieve the list of available versions for provider
nexus.xyz.com/artifatcts/inflobox: could not connect to nexus.xyz.com:
Failed to request discovery document: Get
"https://nexus.xyz.com/.well-known/terraform.json": x509: certificate signed
by unknown authority
Can you please let me know what i am missing.
i was going through this https://github.com/hashicorp/terraform/issues/25485 , since issue is similar to me . Looks like i am also having similar issue after following and applying this https://www.terraform.io/upgrade-guides/0-13.html#new-filesystem-layout-for-local-copies-of-providers.
Looks like that issue is not resolved. Hope you will provide fix for this issue.
Thanks,
shobhan.
Hi @babuamudala,
It seems like for some reason the automatic detection of the local plugin isn’t working for you. I’m not sure why that is from what we’ve discussed so far though, so I think it’ll take some further debugging to understand what’s going on.
One way I can think of to hopefully learn some more about what’s happening on your system is to temporarily set an explicit provider installation configuration that matches the situation that Terraform ought to be detecting automatically, and then we can hopefully learn from how that behaves why the automatic mode isn’t working on your system.
Since you seem to be running Terraform as the user root, the CLI configuration file for that user would be /root/.terraformrc. Please create that file and put the following configuration into it, which is intended to match the intended automatic configuration for your situation:
provider_installation {
filesystem_mirror {
path = "/root/.terraform.d/plugins"
include = ["nexus.xyz.com/artifacts/*"]
}
direct {
exclude = ["nexus.xyz.com/artifacts/*"]
}
}
The meaning of the above is that Terraform should look in the local directory for any provider that belongs to the namespace nexus.xyz.com/artifacts/ but should use upstream registries for any other.
If you run terraform init again with that configuration in place, I expect you will see a different result. That result might be that it now works, because we’ve bypassed the automatic detection behavior and just told Terraform where to look. Alternatively, the result might be that it now fails with a different error message.
Either way, it would be helpful if you could share the trace log output again with this in place. Now that we’re using explicit configuration the relevant part of the log will be later, so I’d ask if you could share the whole trace log so I can see how Terraform is reacting to the explicit configuration.
(Once you’ve tried this you can remove that .terraformrc file to recover the previous behavior. I’m not intending this as a solution to the problem, just as a way to help debug what’s happening.)
Thank you for your response.
I tried above synatax as you recommended but getting same error
2020/10/15 22:03:18 [INFO] Terraform version: 0.13.4
2020/10/15 22:03:18 [INFO] Go runtime version: go1.14.7
2020/10/15 22:03:18 [INFO] CLI args: []string{"/var/terraform13/terraform13", "init"}
2020/10/15 22:03:18 [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2020/10/15 22:03:18 Loading CLI configuration from /root/.terraformrc
2020/10/15 22:03:18 [DEBUG] checking for credentials in "/root/.terraform.d/plugins"
2020/10/15 22:03:18 [DEBUG] Explicit provider installation configuration is set
2020/10/15 22:03:18 [TRACE] Selected provider installation method cliconfig.ProviderInstallationFilesystemMirror("/root/.terraform.d/plugins") with includes [nexus.xyz.com/artifacts/*] and excludes []
2020/10/15 22:03:18 [TRACE] Selected provider installation method cliconfig.ProviderInstallationDirect with includes [] and excludes [nexus.xyz.com/artifacts/*]
2020/10/15 22:03:18 [INFO] CLI command args: []string{"init"}
Error: Could not load plugin
Plugin reinitialization required. Please run "terraform init".
Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.
Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".
3 problems:
- Failed to instantiate provider "nexus.xyz.com/artifacts/inflobox" to
obtain schema: unknown provider "nexus.xyz.com/artifacts/inflobox"
- Failed to instantiate provider "registry.terraform.io/hashicorp/infoblox" to
obtain schema: unknown provider "registry.terraform.io/hashicorp/infoblox"
Thanks,
shobhan
Below error without this TF_LOG=TRACE command
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
nexus.xyz.com/artifacts/inflobox: provider
nexus.xyz.com/artifacts/inflobox was not found in any of the search
locations
Thanks
@apparentlymart
I’m getting same Logs as @babuamudala for diff provider.
JFI:
I’m revisiting my old project setup(Terraform v0.13.0) of inhouse provider after 2 months (which was working back then) but, currently getting this error on terraform init
Is there any change in terraform’s working with the latest update? (Terraform v0.13.5)
I’ve also tried with the -plugin-dir option but still terraform is not able to detect my local binary
Hi @Mohitp98!
Could you also try the explicit CLI configuration like I mentioned in an earlier comment (adjusted to use your own home directory path instead of /root/.terraform.d) and share the same sort of terraform init log output?
I’m still not really sure what’s going on but I’m hoping that having the full log output with the explicit configuration will give a clue.
My suspicion is that the directory structure in the .terraform.d/plugins directory doesn’t match what Terraform is expecting, but it seems to be different enough from Terraform’s expectations that Terraform isn’t recognizing anything in there at all. If you could also share the full directory tree under /home/user/.terraform.d/plugins that might help to understand what isn’t matching Terraform’s expected directory structure.
Thanks, @apparentlymart for the response!
here is a detailed log of explicit installation.
Can you share what exact directory structure is required for terraform to place our provider plugin?
Hey @apparentlymart, Thank you for your help again!
I’ve resolved this issue now.
As I was not aware of the new file system layout effective from Terraform v0.13
It was due to the wrong directory structure created for the local plugin.
Correct Approach:
Under this directory, I’ve placed my provider plugin as follow:
> [user@tmpl-centOs7 plugins]$ pwd
> /home/user/.terraform.d/plugins
> [user@tmpl-centOs7 plugins]$ tree
> .
> `-- local.terraform.com
> `-- user
> `-- ad
> `-- 1.0.0
> `-- linux_amd64
> `-- terraform-provider-ad_v1.0.0
And Inside my HCL config, I’ve mentioned this registry
Terraform will look into this directory for plugin:
terraform { required_providers { ad = { source = "local.terraform.com/user/ad" version = "1.0.0" } } }
Official documentation should highlight these major changes for developers as you have explained here!
Thank you again!
I couldn’t get it to work with /.terraform.d. I had to use /.terraform.