Error reading wrappers from "recovery_kms_hcl"

We’re trying to deploy Boundary from our CI/CD tool hosted on EC2 running in Docker basically. So far all of the attempts end up with the following error:

│ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6: no such file or directory
│   with provider[""],
│   on line 40, in provider "boundary":
│   40: provider "boundary" {
2023-07-27T12:56:16.883Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-07-27T12:56:16.884Z [DEBUG] provider: plugin process exited: path=.terraform/providers/ pid=107
2023-07-27T12:56:16.884Z [DEBUG] provider: plugin exited

This is what I found in the logs:

2023-07-27T12:56:16.818Z [DEBUG] provider.terraform-provider-boundary_v1.1.9_x5: 2023-07-27T12:56:16.818Z [INFO]  plugin: configuring client automatic mTLS
2023-07-27T12:56:16.826Z [DEBUG] provider.terraform-provider-boundary_v1.1.9_x5: 2023-07-27T12:56:16.826Z [DEBUG] plugin: starting plugin: path=/tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6 args=["/tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6"]
2023-07-27T12:56:16.829Z [TRACE] provider.terraform-provider-boundary_v1.1.9_x5: Called downstream: tf_provider_addr=provider tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 tf_rpc=Configure @module=sdk.helper_schema timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.829Z [TRACE] provider.terraform-provider-boundary_v1.1.9_x5: Received downstream response: tf_proto_version=5.3 tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 diagnostic_warning_count=0 @module=sdk.proto diagnostic_error_count=1 tf_provider_addr=provider tf_req_duration_ms=250 tf_rpc=Configure timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.829Z [ERROR] provider.terraform-provider-boundary_v1.1.9_x5: Response contains error diagnostic: tf_provider_addr=provider tf_rpc=Configure @module=sdk.proto diagnostic_summary="error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6: no such file or directory" tf_proto_version=5.3 tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 diagnostic_detail= diagnostic_severity=ERROR timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.830Z [TRACE] provider.terraform-provider-boundary_v1.1.9_x5: Served request: @module=sdk.proto tf_rpc=Configure tf_req_id=229c2805-e686-0a5a-b83d-e7a4254d2b62 tf_proto_version=5.3 tf_provider_addr=provider timestamp=2023-07-27T12:56:16.829Z
2023-07-27T12:56:16.830Z [ERROR] vertex "provider[\"\"]" error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/3756884590/boundary-plugin-kms-awskms-TEwQ6: no such file or directory
2023-07-27T12:56:16.830Z [TRACE] vertex "provider[\"\"]": visit complete, with errors

Here’s the provider config:

terraform {
  required_version = "~> 1.5.0"
  required_providers {
    boundary = {
      source  = "hashicorp/boundary"
      version = "1.1.9"
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.67.0"
provider "boundary" {
  addr             = local.url
  recovery_kms_hcl = <<EOT
    kms "awskms" {
        purpose    = "recovery"
        region = "${local.region}"
        kms_key_id = "${}"
        access_key = "${data.aws_ssm_parameter.iam_access_key.value}"
        secret_key = "${data.aws_ssm_parameter.iam_secret_access_key.value}"

The same configuration works when executing from my local machine so it seems specific to the EC2 environment, unfortunately I just can’t figure out what it is.
At first I thought perhaps it’s some permissions issue and verified that the EC2 instance is able to fetch and encrypt with the KMS key. Then I thought perhaps it’s something to do with Docker and tried executing it locally from Docker without any issues.
I also found this GitHub issue [bug] terraform plan fails with error reading wrappers from "recovery_kms_hcl" on v1.1.5 · Issue #384 · hashicorp/terraform-provider-boundary · GitHub and tried downgrading the provider without any success, also checked the permissions to /tmp as suggested.

The error message is rather vague and we’re running out of ideas here any help is greatly appreciated.

You mentioned that it works on your local machine – are you running Terraform in Docker there as well?

Forgot to mention that the official Terraform Docker image was used. And yes, tested with Docker on local machine also.

I was actually able to find a workaround for this issue by using Ubuntu 22.04 image.
Would still like to get to the bottom of this though.

So the Terraform Docker image worked with this code on Ubuntu 22.04 – what was the other distro/version you were using before that it didn’t work on?

Terraform Docker image (latest) worked locally, it didn’t work in our CICD. In our CICD I got it working by using Ubuntu 22.04 Docker image instead and installing TF on it.

This is a little bit of a shot in the dark, but what happens if instead of the Ubuntu 22.04 image, you use Alpine?