Vault AWS IAM role arn does not work with wilcard with Vault Lambda Layer

Vault
1

Hey Everyone!
We have an issue that is a real blocker for us in terms of scale.
We have AWS Lambdas that use Vault Layer to authenticate with Vault and read secrets from it. (https://developer.hashicorp.com/vault/docs/platform/aws/lambda-extension)
We use Terraform to generate an AWS vault role with an assigned AWS role arn to allow lambda to authenticate.
It works like this:

  1. The developer created AWS Role in AWS. Let’s say the role name will be: hospital-role
  2. Then we add this role to vault with terraform:
resource "vault_aws_auth_backend_role" "hospital-role" {
    backend                         = vault_auth_backend.aws_dev_auth.path
    role                            = "hospital-role"
    auth_type                       = "iam"
    bound_iam_principal_arns        = ["arn:aws:sts::***:role/hospital-role"]
    resolve_aws_unique_ids          = false
    token_ttl                       = var.lambda_default_token_ttl
    token_max_ttl                   = var.lambda_default_max_token_ttl
    token_policies                  = ["hospital-role"]
}

After lambda is deployed it is authenticated and everything works perfectly.
The problem is that we have multiple service/ lambdas per service, which means creating a role per lambda. It is a lot of roles. The way to simplify it is to assign one role to multiple lambdas.
This can be done with a feature that Vault supports, which is role arn wildcard.
https://registry.terraform.io/providers/hashicorp/vault/latest/docs/resources/aws_auth_backend_role
The Docs says:

Wildcards are supported at the end of the ARN, e.g., “arn:aws:iam::123456789012:" will match any IAM principal in the AWS account 123456789012. Wildcards are supported at the end of the ARN, e.g., "arn:aws:iam::123456789012:role/” will match all roles in the AWS account…

These are all examples that I found. So back to our role.
We have role:

resource "vault_aws_auth_backend_role" "CDKRoleDev" {
    backend                         = vault_auth_backend.aws_dev_auth.path
    role                            = "CDKRoleDev"
    auth_type                       = "iam"
    bound_iam_principal_arns        = ["arn:aws:sts::***:role/CDKRoleDev-*"]
    resolve_aws_unique_ids          = false
    token_ttl                       = var.lambda_default_token_ttl
    token_max_ttl                   = var.lambda_default_max_token_ttl
    token_policies                  = ["CDKRoleDev"]
}

And we have a real cloud role attached to lambda.
CDKRoleDev-PLACEHOLDERPlaceholder…
After Lambda starts it throws an error:

| * entry for role CDKRoleDev-PLACEHOLDERPlaceholder... not found

Screenshot of the same error for a different role.

vault
vault1542×455 55.9 KB

We tried tons of times and different namings and conditions, and none of them worked. All of them throw the same error. And yes account reference in the role is correct, we checked :slight_smile:
If we use use regular role without a wildcard it also works. So it is not a connection issue overall.
If somebody has any guesses on what could be an issue, really appreciate your help.
Thanks!

2

Hello,

I don’t have an environment set up right now to test, but to confirm where you are at if you set the Terraform config with

bound_iam_principal_arns = ["arn:aws:sts::***:role/CDKRoleDev-*"]

You get the error

entry for role CDKRoleDev-PLACEHOLDERPlaceholder... not found

But if you set Terraform as

bound_iam_principal_arns = ["arn:aws:sts::***:role/CDKRoleDev-PLACEHOLDERPlaceholder"]

It works?

If that understanding is correct, until I get a chance to set up an environment to test, have you tried removing the - from the ARN? Something like

bound_iam_principal_arns = ["arn:aws:sts::***:role/CDKRoleDev*"]

3

Also, have you seen this support doc? It seems like you may have covered these scenarios but wanted to share:

4

We can close it)
Looks like resolved.
Keep an eye on ARN:

arn:aws:sts::***:role/CDKRoleDev-* - WRONG

arn:aws:iam::***:role/CDKRoleDev-* - CORRECT

Instead of sts, there should be iam :smiley:

1 Like
5

jonathanfrappier Thank you for your help anyway!

1 Like
6

You’re welcome - glad you got it figured out.