Cross-Account KMS?

Suppose the Controller node was in a separate account than the Worker nodes. How do I allow the worker to get validated by the controller when using AWS KMS?

Ideally I’d like to use a cross-account assumable IAM Role between instances (instead of passing in long-lived user credentials to a config file).

How do I go about doing this?

I see that in the awskms config block, there are options for access_key, secret_key, and session_token, however there is no option for an assume_role arn functionality unlike Terraform’s AWS provider.

Passing in an Access_key, Secret_key, and Session_token I think would be impossible since the session token will eventually expire and when a node needs to re-launch itself (through k8’s or autoscaling groups) these credentials would be invalid. Furthermore, I do not know if Boundary requires multiple authentication checks throughout a workers lifetime (and thus if the credentials expire, the worker fails).

I believe if you assign a role with the necessary IAM permissions to the instance the worker is on, Boundary can pick up and use the role credentials automatically, they won’t need to be specified in the worker config.

Correct, see this page, and specifically:

The client uses the official AWS SDK and will use the specified credentials, environment credentials, shared file credentials, or IAM role/ECS task credentials in that order, if the above AWS specific values are not provided.

But does it actually use the shared file credentials?

For example, I have two accounts, AAAAAAAAAAAA and BBBBBBBBBBBB. I have a KMS key in account AAAAAAAAAAAA and an IAM role that account BBBBBBBBBBBB will assume.

I put a shared file onto my worker instance with the terraform provisioners before activating the boundary service

provisioner "remote-exec" {
  inline = [
    "mkdir ~/.aws/"
  ]
}

provisioner "file" {
  destination = "~/.aws/config"
  content = <<EOF
[profile default]
role_arn = arn:aws:iam::AAAAAAAAAAAA:role/boundary-kms-role
credential_source = Ec2InstanceMetadata
region = us-east-1
EOF
}

After boundary is installed onto the client machine, I SSH in and see in the logs thiss message:

Error parsing KMS configuration: error fetching AWS KMS wrapping key information: NotFoundException: Key 'arn:aws:kms:us-east-1:BBBBBBBBBBBB:key/05e633a3-ee51-4345-8c4f-19b7e525cfaf' does not exist

(I would get this same error if I tried doing the call without the shared credentials file)

But when ssh’ing into the machine and performing a call to aws sts-get-caller-identity, I get this output:

{ "UserId": "AROAS3K5LVA7JGKWUPBPC:botocore-session-1629130916", "Account": "AAAAAAAAAAAA", "Arn": "arn:aws:sts::AAAAAAAAAAAA:assumed-role/boundary-test/botocore-session-1629130916" }

I’m convinced that it’s not actually using the shared file credentials, but I might be wrong.

But I just found out that there’s a way to share AWS KMS keys without needing to assume roles by sharing the AWS KMS resource directly, so I’ll try that out in the meanwhile.

According to the docs from the official Go SDK for AWS, it expects the credentials file to be at ~/.aws/credentials.

Some other comments:

  • There are some env vars that can be set related to some STS settings. These are not currently exposed in configuration.
  • Currently the credentials file location cannot be overridden, and neither can the profile, which means it will use the default profile; from the AWS SDK docs "AWS_PROFILE" or "default" if the environment variable is not set

Tried that too. Added this little block as well.

provisioner "file" {
    destination = "~/.aws/credentials"
    content = <<EOF
[default]
role_arn = ${var.kms_role_arn}
credential_source = Ec2InstanceMetadata
region = ${var.kms_region}
EOF
}

Same result

Setting “AWS_ROLE_ARN” does look promising, however, I’ve found an alternative solution that works much easier than cross account assumerole for now.

Sharing the KMS with external accounts is much simpler and requires no additional instance config. For those who might stumble upon the same problem in the future working from the boundary reference architecture as a base, add these edits into your code:

  1. Added the desired target accounts as a variable into your config:
variable "shared_accounts" {
  type = list(string)
}
  1. Edit the worker_auth KMS key to have a policy to share with external accounts
resource "aws_kms_key" "worker_auth" {
  description             = "Boundary worker authentication key"
  deletion_window_in_days = 10

  tags = {
    Name = "${var.tag}-${random_pet.test.id}"
  }

#Note: "kms:ListKeys", "kms:ListAliases" aren't included because they aren't part of shared key permissions
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }%{for account_num in var.shared_accounts},
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:${data.aws_partition.current.partition}:iam::${account_num}:root"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:GenerateDataKey",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
        %{endfor}
    ]
}
EOF
}
  1. Save the key ARN
output "kms_key_id" {
  value = aws_kms_key.worker_auth.arn
}
  1. In the account that is using the shared KMS, make sure the instance has a role policy that grants the same essential permissions onto that shared key arn like those needed in the original architecture. i.e., supposing the key arn is var.kms_key_id
resource "aws_iam_role_policy" "boundary" {
  name = "${var.tag}-${random_pet.test.id}"
  role = aws_iam_role.boundary.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:DescribeKey",
      "kms:GenerateDataKey",
      "kms:Decrypt",
      "kms:Encrypt",
      "kms:ListKeys",
      "kms:ListAliases"
    ],
    "Resource": [
      "${var.kms_key_id}"
    ]
  }
}
EOF
}
  1. MOST IMPORTANT: In the worker config, make sure the key id passed in is indeed the ARN to the key, and not just the long 32-character alphanumeric key-id.
    i.e. Make sure in this config:
%{ if kms_type == "aws" }
kms "awskms" {
	purpose    = "worker-auth"
	key_id     = "global_root"
  kms_key_id = "${kms_worker_auth_key_id}"
}
%{ else }
kms "aead" {
	purpose = "worker-auth"
	aead_type = "aes-gcm"
	key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
	key_id = "global_worker-auth"
}
%{ endif }

kms_key_id is an ARN, such as the output from the code above.

1 Like

I’m glad you found something that works for you! I’m also doing a series of commits across libs that will allow some of the values you need to be configured directly via the KMS block (first PR is at Allow configuration of some parameters via config object by jefferai · Pull Request #6 · hashicorp/go-secure-stdlib · G). Hopefully this should allow more configuration starting with the first release after 0.5.1.

1 Like

Will this functionality be ported into the terraform provider as well?

Currently there’s a recovery_kms_hcl argument block that can be used to authenticate into boundary through the kms like so:

provider "boundary" {
  addr             = var.url
  recovery_kms_hcl = <<EOT
kms "awskms" {
	purpose    = "recovery"
	key_id     = "global_root"
  kms_key_id = "${var.kms_recovery_key_id}"
}
EOT
}

I haven’t experienced this yet firsthand, but I imagine if the default AWS profile was different from that of where the KMS was accessible, there would probably be definite problems trying to access the key (since there’s no arguments for aws_profile or role_arn).

It can be! It will take a bit for it to filter through new releases of things but I’ll make sure a PR goes in to update the related dependencies. PR two is Expose more credential config parameters by jefferai · Pull Request #38 · hashicorp/go-kms-wrapping · GitHub

1 Like

These changes will be in 0.5.1 out today or tomorrow – see https://github.com/hashicorp/boundary/pull/1468/files#diff-b9e576dcaa6dce04299e8387aca5354c0c226d018b7d0dc8f22d0b22fd4c132aR56 for the docs that will be up on the website.

If those parameters aren’t sufficient, let me know and we can see what can be added later. (These commits were just exposing existing parameters in our lib to the configuration of

1 Like