EKS 1.22 & HCP Vault with transit key

Vault
1

Hi,

We are testing HCP managed vault. Use case is (EKS) Cluster to HCP Vault on AWS. For now we are tesing public adress in HCP Vault.
Followed the tutorial HCP Vault with Amazon Elastic Kubernetes Service | Vault - HashiCorp Learn .

We have also tested the same configuration against open source vault cluster in our eks cluster, and there it works…:slight_smile:

We have enabled:
1: vault auth enable Kubernetes
2: create auth/kubernetes/config as dokumentet in tutorial
3: vault secrets enable transit
4: vault write -f transit/keys/test
5. created the following policy test

  path "transit/encrypt/test" {
  capabilities = [ "update" ]
  }
  path "transit/decrypt/test" {
  capabilities = [ "update" ]
  }

6: created service account vault-auth in namespace (EKS)

7:

vault write auth/kubernetes/role/test bound_service_account_names=vault-auth bound_service_account_namespaces=*** policies=test ttl=24h

We test the policy with login to admin in HCP:

$ vault write transit/encrypt/transit1 plaintext=$(base64 <<< "some message")
Key            Value
---            -----
ciphertext     vault:v1:VaMZu3aXDSZd/S2gv6qgXi8YnzYSlA7/emQFPePRwvpeWfMBaItVjAw=
key_version    1

Test case 1:

We have connections from our eks cluster to HCP Vault with auth kubernetes token, so there everything works.


# curl -GET http://127.0.0.1:8200/v1/auth/token/lookup-self | jq -r ".data"
 
{
  "creation_time": 1654091353,
  "creation_ttl": 86400,
  "display_name": "admin-auth-kubernetes-test-vault-auth",
  "entity_id": "***..2d09d5c3",
  "expire_time": "2022-06-02T13:49:14.686409588Z",
  "explicit_max_ttl": 0,
  "issue_time": "2022-06-01T13:49:13.051170384Z",
  "last_renewal": "2022-06-01T13:49:14.686409708Z",
  "last_renewal_time": 1654091354,
  "meta": {
    "role": "test",
    "service_account_name": "vault-auth",
    "service_account_namespace": "namespace",
    "service_account_secret_name": "",
    "service_account_uid": "cef0a8af-f279-*****"
  },
  "namespace_path": "admin/,
  ............

Test case 2:

We create a new policy for transit in order to give access to the key, but when trying to use the key we get a permission denied.

Polyci created, as per documentation the following policy should work:
But we always got permission denied agianst HCP vault.

path "transit/*" { capabilities = [ "update" ] }
 
path "transit/encrypt/test" {
  capabilities = [ "update" ]
}
path "transit/decrypt/test" {
  capabilities = [ "update" ]
}

# curl -s request POST --data '{"plaintext": "'`echo ***** | base64`'"}' http://127.0.0.1:8200/v1/transit/encrypt/test | jq .
{
  "errors": [
    "1 error occurred:\n\t* permission denied\n\n"
2

You didn’t mention the version of your Vault. There was a breaking change in Kubernetes as of 1.21+.

3

We use HCP Vault hosted version of Vault, which is operated by HashiCorp (Vault version v1.10.3)

4

Please could you edit your post, putting each copy/paste into a code block, so it isn’t mangled by having special characters interpreted as formatting?

It’s pretty hard to reverse engineer right now.

5

Hope it is better now, thanks for the feedback :+1:

6

There are some odd errors in the syntax of your curl commands - are you retyping them from memory with some errors, or something like that?

I can think of two possible problems here:

  1. You may not have correctly configured your policy… However you have truncated your auth/token/lookup-self output so we can’t confirm this.

  2. You may not be specifying the Vault namespace correctly.

7

You may be missing the “admin” namespace which all HCP Vault clusters use by default.

curl --header "X-Vault-Namespace: admin"

I would attempt that outside of k8s first.

8

This works for me when VAULT_TOKEN, VAULT_ADDR, and VAULT_NAMESPACE are all set:

curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request PUT \
    --data '{"plaintext": "'`echo test-message | base64`'"}' \
    $VAULT_ADDR/v1/transit/encrypt/test | jq
9

Thanks!

"X-Vault-Namespace: $VAULT_NAMESPACE"

was the issue for us, when we use vault namespaces we can use curl from eks to HCP env*
Thanks for the help!