We’re trying to deploy Boundary from our CI/CD tool hosted on EC2 running in Docker basically. So far all of the attempts end up with the following error:
╷
│ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6: no such file or directory
│
│ with provider["registry.terraform.io/hashicorp/boundary"],
│ on providers.tf line 40, in provider "boundary":
│ 40: provider "boundary" {
│
╵
2023-07-27T12:56:16.883Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-07-27T12:56:16.884Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/boundary/1.1.9/linux_amd64/terraform-provider-boundary_v1.1.9_x5 pid=107
2023-07-27T12:56:16.884Z [DEBUG] provider: plugin exited
This is what I found in the logs:
2023-07-27T12:56:16.818Z [DEBUG] provider.terraform-provider-boundary_v1.1.9_x5: 2023-07-27T12:56:16.818Z [INFO] plugin: configuring client automatic mTLS
2023-07-27T12:56:16.826Z [DEBUG] provider.terraform-provider-boundary_v1.1.9_x5: 2023-07-27T12:56:16.826Z [DEBUG] plugin: starting plugin: path=/tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6 args=["/tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6"]
2023-07-27T12:56:16.829Z [TRACE] provider.terraform-provider-boundary_v1.1.9_x5: Called downstream: tf_provider_addr=provider tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 tf_rpc=Configure @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:558 @module=sdk.helper_schema timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.829Z [TRACE] provider.terraform-provider-boundary_v1.1.9_x5: Received downstream response: tf_proto_version=5.3 tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tf5serverlogging/downstream_request.go:37 diagnostic_warning_count=0 @module=sdk.proto diagnostic_error_count=1 tf_provider_addr=provider tf_req_duration_ms=250 tf_rpc=Configure timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.829Z [ERROR] provider.terraform-provider-boundary_v1.1.9_x5: Response contains error diagnostic: tf_provider_addr=provider tf_rpc=Configure @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_summary="error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6: no such file or directory" tf_proto_version=5.3 tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 diagnostic_detail= diagnostic_severity=ERROR timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.830Z [TRACE] provider.terraform-provider-boundary_v1.1.9_x5: Served request: @module=sdk.proto tf_rpc=Configure tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:567 tf_proto_version=5.3 tf_provider_addr=provider timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.830Z [ERROR] vertex "provider[\"registry.terraform.io/hashicorp/boundary\"]" error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6: no such file or directory
2023-07-27T12:56:16.830Z [TRACE] vertex "provider[\"registry.terraform.io/hashicorp/boundary\"]": visit complete, with errors
Here’s the provider config:
terraform {
required_version = "~> 1.5.0"
required_providers {
boundary = {
source = "hashicorp/boundary"
version = "1.1.9"
}
aws = {
source = "hashicorp/aws"
version = "~> 4.67.0"
}
}
...
provider "boundary" {
addr = local.url
recovery_kms_hcl = <<EOT
kms "awskms" {
purpose = "recovery"
region = "${local.region}"
kms_key_id = "${data.aws_kms_key.by_alias.id}"
access_key = "${data.aws_ssm_parameter.iam_access_key.value}"
secret_key = "${data.aws_ssm_parameter.iam_secret_access_key.value}"
}
EOT
}
The same configuration works when executing from my local machine so it seems specific to the EC2 environment, unfortunately I just can’t figure out what it is.
At first I thought perhaps it’s some permissions issue and verified that the EC2 instance is able to fetch and encrypt with the KMS key. Then I thought perhaps it’s something to do with Docker and tried executing it locally from Docker without any issues.
I also found this GitHub issue [bug] terraform plan fails with error reading wrappers from "recovery_kms_hcl" on v1.1.5 · Issue #384 · hashicorp/terraform-provider-boundary · GitHub and tried downgrading the provider without any success, also checked the permissions to /tmp as suggested.
The error message is rather vague and we’re running out of ideas here any help is greatly appreciated.