Vault Secrets Operator. Permissions Denied on Vault login

Hi guys, I am attempting to setup Vault Secrets Operator with Kubernetes auth with my External SASS Vault. VSO gets a 403 on login against my public vault. It is setup as follows:

vault secrets enable -path=kvv2 kv-v2
vault kv put kvv2/webapp username="web-user" password=":pa55word:"

vault auth enable -path=vso kubernetes

vault policy write webapp-ro - <<EOF

path "kvv2/data/webapp" {

capabilities = ["read"]

}

path "kvv2/metadata/webapp" {

capabilities = ["read"]

}

EOF
TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth --output='go-template={{ .data.token }}' | base64 --decode)

KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
`vault write auth/vso/config kubernetes_host="$KUBE_HOST" \
disable_local_ca_jwt="true" \
kubernetes_ca_cert="$KUBE_CA_CERT"`

This previously included a token_reviewer_jwt but this topic suggested otherwise.

vault write auth/vso/role/vso-role \
      bound_service_account_names=vault-auth \
      bound_service_account_namespaces=* \
      policies=webapp-ro \
      audience=vault \
      ttl=24h

This previously included a namspace=default but this topic suggested otherwise.

On K8S ,the service account:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

Then the VSO connection and secret map:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  # address to the Vault server.
  address: https://<public URI>.hashicorp.cloud:8200
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: vso
  kubernetes:
    role: vso-role
    serviceAccount: default
    audiences:
      - vault
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
spec:
  vaultAuthRef: vault-auth
  mount: kvv2
  type: kv-v2
  path:  webapp
  refreshAfter: 300s
  destination:
    create: true
    name: vso-handled

Finally I can test as follows against vault:
vault write auth/vso/login role=vso-role jwt=$TOKEN_REVIEW_JWT

But VSO gets 403. Your help is much appreciated.

If you are using HCP Vault with the default config, have you tried prepending admin to the vaultConnectionRef? HCP Vault runs everything from the admin namespace, instead of root like a self-hosted cluster would.

Example:

spec:
  vaultConnectionRef: admin/vault-connection

This appears to be a gap in our documentation related to using Vault Namespaces (Enterprise/HCP feature) so if this works I can update our docs.

Basing the suggestion on this: https://github.com/hashicorp/vault-secrets-operator/blob/main/api/v1beta1/vaultauth_types.go#L119

Unfortunately, I get “vault-connection not found” when I

specify:

spec:
  vaultConnectionRef: admin/vault-connection

Could there be a change I am suppose to apply to my VaultConnection?

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  # address to the Vault server.
  address: https://<plubic vautl uri>.hashicorp.cloud:8200
---

Hi - thanks for trying it out (I am trying to wrap up something in progress so i can setup m y local environment to test this.

What if you set up your spec like this?

spec:
  # address to the Vault server.
  address: https://<plubic vautl uri>.hashicorp.cloud:8200
  namespace: admin

Thanks again for trying, making suggestions based on reading that file above (also posted a question internally to try and track down the answer)

Thank you for the suggestion but I get this error " unknown field spec.namespace".

If I do the following:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: vso
  kubernetes:
    role: vso-role
    serviceAccount: default
    audiences:
      - vault
  namespace: "admin" #This is new, see ref below.

Then I get

Namespace: admin
URL: PUT https://vault-public-vault-…hashicorp.cloud:8200/v1/auth/vso/login
Code: 403. Errors:

  • service account name not authorized {“type”: “Warning”, “object”: {“kind”:“VaultStaticSecret”,“namespace”:“default”,“name”:“vault-static-secret”,“uid”:“…”,“apiVersion”:“secrets.hashicorp.com/v1beta1",“resourceVersion”:"1447444”}, “reason”: “VaultClientConfigError”}

Ref article

# Vault namespace where the auth backend is mounted (requires Vault Enterprise)
  # namespace: ""

Thanks again for trying out the suggestion. That most recent error looks like you are now getting through to Vault at least. I will try to set up my local environment and do some more testing.

Not 100% set up yet, but someone just pointed out to me that you also need it in the secret spec

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
spec:
  vaultAuthRef: vault-auth
  namespace: "admin"
  mount: kvv2
  type: kv-v2
  path:  webapp
  refreshAfter: 300s
  destination:
    create: true
    name: vso-handled

Assuming of course this is all set up in the admin namespace.

Thanks, I still get the 403 but the error message is clear “service account name not authorized”

I will just highlight the areas that I think are important:

vault write auth/vso/config \
      kubernetes_host="$KUBE_HOST" \
      disable_local_ca_jwt="true" \
      kubernetes_ca_cert="$KUBE_CA_CERT"
vault write auth/vso/role/vso-role \
      bound_service_account_names=vault-auth \
      bound_service_account_namespaces=* \
      policies=webapp-ro \
      ttl=24h \
      audience=vault
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  # address to the Vault server.
  address: https://vault-public-vault-.....hashicorp.cloud:8200
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: vso
  kubernetes:
    role: vso-role
    serviceAccount: default # vault-auth
    audiences:
      - vault
  namespace: "admin" #Enterprise Vault only
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
spec:
  vaultAuthRef: vault-auth
  namespace: "admin" #Enterprise Vault only
  mount: kvv2
  type: kv-v2
  path:  webapp
#  version: 2
  refreshAfter: 300s
  destination:
    create: true
    name: vso-handled

Service Account:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

Thank you for your help.

Hello,

Just wanted to confirm I have not dropped this, just need the time to set up my environment. I hope to get to that to start my day tomorrow.

Thank you Jonathan, I got it working with your advice! The key here is the namespace in VaultAuth and VaultStaticSecret in the k8s yaml. Also the vault config and vault oath role bound namespace.

Vault Config:
vault auth enable -path=vso kubernetes

vault write auth/vso/config \
      kubernetes_host="$KUBE_HOST" \
      kubernetes_ca_cert="$KUBE_CA_CERT"
vault write auth/vso/role/vso-role \
      bound_service_account_names=vault-auth \
      bound_service_account_namespaces=default \
      policies=webapp-ro \
      ttl=24h

Cluster Config:

---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
 name: vault-auth
 annotations:
   kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: role-tokenreview-binding
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:auth-delegator
subjects:
 - kind: ServiceAccount
   name: vault-auth
   namespace: default
#kubectl apply -f sa.yaml
#kubectl delete -f sa.yaml
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  # address to the Vault server.
  address: https://vault-public-vault-.....z1.hashicorp.cloud:8200
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: vso
  kubernetes:
    role: vso-role
    serviceAccount: vault-auth
  namespace: "admin" #Enterprise Vault only
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
spec:
  vaultAuthRef: vault-auth
  namespace: "admin" #Enterprise Vault only
  mount: kvv2
  type: kv-v2
  path:  webapp
  refreshAfter: 300s
  destination:
    create: true
    name: vso-handled
#kubectl apply -f crd.yaml
#kubectl delete -f crd.yaml
#See https://developer.hashicorp.com/vault/docs/platform/k8s/vso/sources/vault
#See VaultStaticSecret - https://developer.hashicorp.com/vault/docs/platform/k8s/vso/examples
2 Likes

Awesome! Nice work! I’m going to take this thread and make sure our docs and tutorials reflect this so others with HCP Vault or Vault Enterprise have the necessary info.

Thanks for working through it with me.

1 Like