Installing custom provider locally error on plan

Terraform Providers Plugin Development
1

Hello, I am following the docs here to install a custom provider called “notary” locally. I get this error as listed below.

│ Error: Failed to load plugin schemas
│ 
│ Error while loading schemas for plugin components: Failed to obtain
│ provider schema: Could not load the schema for provider
│ tf-registry.bluecat.io/edge/notary: failed to instantiate provider
│ "tf-registry.bluecat.io/edge/notary" to obtain schema: fork/exec
│ .terraform/providers/tf-registry.bluecat.io/edge/notary/1.0.0/linux_amd64/terraform-provider-notary_v1.0.0_x5:
│ no such file or directory..

Although the error says no such file or directory.. the provider is there where it should be. I have compared this with downloaded providers and verified that it is listed in the correct location yet the error persists.

I am adding a more verbose logging (after the terraform init steps which succeeds) showing the .terraform.lock.hcl in the hope that some one here can spot what is wrong.

$ ls -l tf-registry.bluecat.io/edge/notary/1.0.0/linux_amd64
total 18628
-rwxrwxr-x    1 root     root      19073055 Oct 26 09:16 terraform-provider-notary_v1.0.0_x5
$ terraform plan -out plan || echo "failed"
╷
│ Error: Failed to load plugin schemas
│ 
│ Error while loading schemas for plugin components: Failed to obtain
│ provider schema: Could not load the schema for provider
│ tf-registry.bluecat.io/edge/notary: failed to instantiate provider
│ "tf-registry.bluecat.io/edge/notary" to obtain schema: fork/exec
│ .terraform/providers/tf-registry.bluecat.io/edge/notary/1.0.0/linux_amd64/terraform-provider-notary_v1.0.0_x5:
│ no such file or directory..
╵
failed
$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/hashicorp/aws" {
  version     = "4.30.0"
  constraints = "~> 4.18"
  hashes = [
    "h1:BFfhRf8my/aa0+YOSJv0xfjLQkToF475TJTMhTZfYec=",
    "h1:IcwEoCZhcgFkk4ySB+6lvEPS90eys+FVuj+TZYVhGS0=",
    "zh:08213f3ba960621448754211f148730edb59194919ee476b0231b769a5355028",
    "zh:29c90d6f8bdae0e1469417ade28fa79c74c2af49593c1e2f24f07bacbca9e2c9",
    "zh:5c6e9fab64ad68de6cd4ec6cbb20b0f75ba1e51a8efaeda3fe65419f096a06cb",
    "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
    "zh:9bf42718580e8c5097227df34e1bfa0a10a23eac9f527d97c2819c163087b402",
    "zh:9f87e42e0f3d145fb0ad4aaff7ddded5720a64f9303956b33bd274c6dd05c05b",
    "zh:bf0519ed9615bc408b72a0aebe1cc075d4c2042325590ba13dd264cd264907ea",
    "zh:c3ac9e1cbd0935614f5a3c9cdb4cf9c6a1045937fe38e61da7c5c0fb7a069870",
    "zh:d0c184476ada38c50acc068214ed1252b4fcf80b6be900fc1aed32cbb49f8ff6",
    "zh:d4987dc7b7a69ea58f2b3ff0ea4ffc1b61a97881dbb8583c9fcf9444b753a6c2",
    "zh:e8037376c81aeb98d8286dc19fba7f8eb053444d4b9484ea6a922382cffc1a85",
    "zh:ecdabb44b48addc8483bca7bd683614a347367ae950ca8b6a6880679f5c12abd",
  ]
}
provider "registry.terraform.io/hashicorp/helm" {
  version = "2.6.0"
  hashes = [
    "h1:QZcB0CGaRloxrq1JjHF4ZLauaoJ8fHF2MsXFezR0COw=",
    "h1:rGVucCeYAqklKupwoLVG5VPQTIkUhO7WGcw3WuHYrm8=",
    "zh:0ac248c28acc1a4fd11bd26a85e48ab78dd6abf0f7ac842bf1cd7edd05ac6cf8",
    "zh:3d32c8deae3740d8c5310136cc11c8afeffc350fbf88afaca0c34a223a5246f5",
    "zh:4055a27489733d19ca7fa2dfce14d323fe99ae9dede7d0fea21ee6db0b9ca74b",
    "zh:58a8ed39653fd4c874a2ecb128eccfa24c94266a00e349fd7fb13e22ad81f381",
    "zh:6c81508044913f25083de132d0ff81d083732aba07c506cc2db05aa0cefcde2c",
    "zh:7db5d18093047bfc4fe597f79610c0a281b21db0d61b0bacb3800585e976f814",
    "zh:8269207b7422db99e7be80a5352d111966c3dfc7eb98511f11c8ff7b2e813456",
    "zh:b1d7ababfb2374e72532308ff442cc906b79256b66b3fe7a98d42c68c4ddf9c5",
    "zh:ca63e226cbdc964a5d63ef21189f059ce45c3fa4a5e972204d6916a9177d2b44",
    "zh:d205a72d60e8cc362943d66f5bcdd6b6aaaa9aab2b89fd83bf6f1978ac0b1e4c",
    "zh:db47dc579a0e68e5bfe3a61f2e950e6e2af82b1f388d1069de014a937962b56a",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}
provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "2.13.1"
  constraints = "~> 2.11"
  hashes = [
    "h1:cN3OwZvhtn/y3XfnGQ4hi+7oZp1gU2zVYhznRv2C7Qg=",
    "h1:pZ+wybkE8QT/rKl7n8wlzR4J7d5YJU92RR3pfbssFvI=",
    "zh:061f6ecbbf9a3c6345b56c28ebc2966a05d8eb02f3ba56beedd66e4ea308e332",
    "zh:2119beeccb35bc5d1392b169f9fc748865261b45fb75fc8f57200e91658837c6",
    "zh:26c29083d0d84fbc2e356e3dd1db3e2dc4139e943acf7a318d3c98f954ac6bd6",
    "zh:2fb5823345ab05b3df74bb5c51c61072637d01b3cddffe3ad36a73b7d5b749e6",
    "zh:3475b4422fffaf58584c4d877f98bfeff075e4a746f13e985d2cb20adc873a6c",
    "zh:366b4bef49932d1d71b12849c1878c254a887962ff915f37982299c1185dd48a",
    "zh:589f9358e4a4bd74a83b97ccc64df455ddfa64c4c4e099aef30fa29080497a8a",
    "zh:7a0d75e0e4fee6cc5599ac9d5e91de563ce9ea7bd8137480c7abd09642a9e72c",
    "zh:a297a42aefe0650e3d9fbe55a3ee48b14bb8bb5edb7068c09512d72afc3d9ca5",
    "zh:b7f83a89b646542d02b733d464e45d6d0739a9dbb921305e7b8347e9fc98a149",
    "zh:d4c721174a598b66bd1b29c40fa7cffafe90bb58186cd7506d792a6b04161103",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}
provider "registry.terraform.io/hashicorp/local" {
  version = "2.2.3"
  hashes = [
    "h1:KmHz81iYgw9Xn2L3Carc2uAzvFZ1XsE7Js3qlVeC77k=",
    "h1:aWp5iSUxBGgPv1UnV5yag9Pb0N+U1I0sZb38AXBFO8A=",
    "zh:04f0978bb3e052707b8e82e46780c371ac1c66b689b4a23bbc2f58865ab7d5c0",
    "zh:6484f1b3e9e3771eb7cc8e8bab8b35f939a55d550b3f4fb2ab141a24269ee6aa",
    "zh:78a56d59a013cb0f7eb1c92815d6eb5cf07f8b5f0ae20b96d049e73db915b238",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:8aa9950f4c4db37239bcb62e19910c49e47043f6c8587e5b0396619923657797",
    "zh:996beea85f9084a725ff0e6473a4594deb5266727c5f56e9c1c7c62ded6addbb",
    "zh:9a7ef7a21f48fabfd145b2e2a4240ca57517ad155017e86a30860d7c0c109de3",
    "zh:a63e70ac052aa25120113bcddd50c1f3cfe61f681a93a50cea5595a4b2cc3e1c",
    "zh:a6e8d46f94108e049ad85dbed60354236dc0b9b5ec8eabe01c4580280a43d3b8",
    "zh:bb112ce7efbfcfa0e65ed97fa245ef348e0fd5bfa5a7e4ab2091a9bd469f0a9e",
    "zh:d7bec0da5c094c6955efed100f3fe22fca8866859f87c025be1760feb174d6d9",
    "zh:fb9f271b72094d07cef8154cd3d50e9aa818a0ea39130bc193132ad7b23076fd",
  ]
}
provider "registry.terraform.io/hashicorp/tls" {
  version     = "4.0.3"
  constraints = "~> 4.0.3"
  hashes = [
    "h1:TtxH+PuCVPoWUpkMhjdz4QB1i7A4/Ag9kIouPabKDrI=",
    "zh:0b34a21c535db27a71d1b76a635352ad5dc31d81cdee3d34926629c8919990a1",
    "zh:160008f7a89ec99f1a79d05bd9c2f11679edbcac5ecf43f618d0a260d003d5ce",
    "zh:2b951a9c75be26ff320d979c121832866db75f08bcbe326cf5290c8497e345a4",
    "zh:5032695a3fb47914fa6037f21e7f06d0b684ffddde0c758477a63008476e97d0",
    "zh:581f518e5104e5bdbdb9f5af35628dcb04a6fee11c6279e4ac1480d7a2bde39f",
    "zh:66f36f9da00e7a39952e9e4cf158b552a98d8afabbb74804bf27b872b4fdefc6",
    "zh:a159589b44d353376f86d05f1819952445f9a1c00a3bcc2745217d6f513f5147",
    "zh:de077f6b852c8d637dbd9b9542ffd590db3d96f9f718acee6645ac6e22365b3c",
    "zh:e0ea922619aea6b97f416fd8c129af944df9f37b81417ae7b970c6895498843e",
    "zh:e7705f4f35332f598e6d4078c7e7e734e4728c5ec8642a50672e29d51d8f3be3",
    "zh:eb64e7a5ebd37b8f59acc91e033897a5d454998f70e57047c94dbfb649503320",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}
provider "registry.terraform.io/hashicorp/vault" {
  version     = "3.9.1"
  constraints = "~> 3.9.1"
  hashes = [
    "h1:AaFthkSQCvOi0Wz/AMrbuYMlIDhk/9h3PuTI2GXt3tU=",
    "zh:0c24d3c8dfece9a3318b6189f74ad226e4ba3ecee143ec850707f289d3e044e8",
    "zh:15612008c7f8058ff1d009af30d2501e239bc78c1a2f4419e792acf6f24482a5",
    "zh:3cf66c144ac77ec1ba287daf5dd45d2a4538355bdc5d12b7f84557c80face58b",
    "zh:4350c29699fad275b3931c00cc09040f8d55424b0e637285dc1bdb119c7957ab",
    "zh:47d22db14337130589dac95851bcb15a7217a379e798e9bba4e52dd4688194da",
    "zh:5ed58f9ac5bd505e14f6753ebce2fa9462ccb4ffd318d3128915562e0ab1f539",
    "zh:6d89d22943200f9c9b3bab70787ee9b171053eeff28c52717068945d9554da92",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:824014d463f72ae0ef1d11c76e39e161f3327381a594b630010b872892a93842",
    "zh:8e5fa87f282c979b9cd96a504644a7cb37d523ffb3a2a4b4014ec113fa64c2b1",
    "zh:93846d4ba7874b0aac9c3f710ba2edcf9b830bab76f50480bf7bddff8cae5445",
    "zh:9ccdab8d862e85a1107c29de0cbe0495a1d374a69985db6f41c10b9bd475682f",
  ]
}
provider "tf-registry.bluecat.io/edge/notary" {
  version     = "1.0.0"
  constraints = ">= 1.0.0"
  hashes = [
    "h1:R958VYfcWOmwklMDiUaBHxwwYqFAKURhbDvZLOPNW9s=",
  ]
}
$ ls -l .terraform/providers/tf-registry.bluecat.io/edge/notary/1.0.0/linux_amd64/
total 18628
-rwxrwxr-x    1 root     root      19073055 Oct 26 09:16 terraform-provider-notary_v1.0.0_x5
$ ls -l .terraform/providers/registry.terraform.io/hashicorp/aws/4.30.0/linux_amd64
total 264520
-rwxrwxr-x    1 root     root     270868480 Oct 26 09:16 terraform-provider-aws_v4.30.0_x5
$ ls -l .terraform/providers/registry.terraform.io/hashicorp/kubernetes/2.13.1/linux_amd64
total 43560
-rwxrwxr-x    1 root     root      44605440 Oct 26 09:16 terraform-provider-kubernetes_v2.13.1_x5
2

As an update, this is still not working after I changed from explicit to implicit filesystem_mirror and this time I have a different error

$ terraform -chdir=$CI_PROJECT_DIR/notary init || echo "Failed"
Initializing the backend...
Initializing provider plugins...
- Finding tf-registry.bluecat.io/edge/notary versions matching "~> 1.0"...
- Installing tf-registry.bluecat.io/edge/notary v1.0.0...
╷
│ Error: Failed to install provider
│ 
│ Error while installing tf-registry.bluecat.io/edge/notary v1.0.0: after
│ installing tf-registry.bluecat.io/edge/notary it is still not detected in
│ the target directory; this is a bug in Terraform
╵
Failed

Just in case you wonder, this does work perfectly fine from my computers (mac and linux) but doesn’t work in a gitlab-ci pipeline with the errors shown here.

3

I eventually tracked the error to the build process. If you build in golang:1.x.x-bullseye and then pull the terraform:1.x.x. docker image to run your binary you get this… build on golang:1.x.x-alpine then we are fine.

2 Likes
4

Hi @yoeluk :wave: Glad to hear that you were able to figure out your issue, but is it possible to show the build process (even if example code)? I’m just wondering if there are larger issues at play that others may run into in their environments. It’d be great to document those and help future travelers. Thanks!