Can not unseal Vault using Service Account IAM Roles in AWS

Vault
1

Hi Everyone,

I have a problem with AWS EKS and IAM Roles. I have created an EKS Cluster for Vault using eksctl. In order to make the Auto Unseal Vault feature work you need to specify a Policy to be able to use AWS KMS. I have already that policy and if I specify that policy in the eksctl cluster yaml in the config then everything works.

iam:
  attachPolicyARNs: ...

The problem is that I want to only attach this policy to the service account that Vault Pod uses. And to do so I execute this two commands and I remove the policy ARN from the cluster config when creating it:

eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --approve
eksctl create iamserviceaccount \
    --name vault \
    --namespace default \
    --cluster $CLUSTER_NAME \
    --attach-policy-arn arn:aws:iam::$ACCOUNT_ID:policy/VaultUnsealingPolicy \
    --approve \
    --override-existing-serviceaccounts

And if I look at the Vault Service Account:

Name:                vault
Namespace:           default
Labels:              app.kubernetes.io/instance=vault
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=vault
                     helm.sh/chart=vault-0.6.0
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::$ACCOUNT_ID:role/eksctl-testing-vault-ha8-addon-iamserviceacc-Role1-H1BPN7HUEFQ2
                     meta.helm.sh/release-name: vault
                     meta.helm.sh/release-namespace: default
Image pull secrets:  <none>
Mountable secrets:   vault-token-p2f9t
Tokens:              vault-token-p2f9t
Events:              <none>

And if I look at that role in the AWS Console it has the Vault Unseal Policy attached.

However, vault pods keep logging this error:

Error parsing Seal configuration: error fetching AWS KMS wrapping key information: AccessDeniedException: User: arn:aws:sts::345070817929:assumed-role/eksctl-testing-vault-ha8-nodegrou-NodeInstanceRole-DAMVREBCR0QG/i-0f8395b93d139b3d9 is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-west-1:345070817929:key/90c696fa-b3b1-41a7-9b52-92ec89880d48
        status code: 400, request id: 75797544-732c-4bbc-b191-b9861fa30333

So it keeps saying that the Worker Node can not get KMS Key, but instead I configured the service account to do it.

Is it something I am missing about roles?

Thanks!

2

I think this feature https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html was added in AWS Go SDK in about the middle of 2019. So maybe Vault software needs to be updated (go module pointing to AWS Go SDK) in order to have it working.

Has anyone tried?

Thanks!

3

This would be nice getting to work. I’m only having success giving the EKS instance role access to the KMS key. It doesn’t seem to care about the serviceaccount annotation if it’s set.

Thank you!

4

I ran into a similar issue and solved it by adding a CA Thumbprint to the AWS Identity Provider. Creating one through the AWS Console automatically adds it, however, using the API it does not.

Here is the page that help me through the problem: https://github.com/terraform-providers/terraform-provider-aws/issues/10104