I’m using the Vault Secrets Operator to get a Vault Static Secret from a kv2 secrets engine using short lived service tokens and TokenReview API in a clean k0s cluster with a clean external vault installation using various tutorials including the official one. I have been getting this error in all cases that I was unable to work around until now:
auth.kubernetes.auth_kubernetes_b1a0b099: login unauthorized: err=“lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token”
I was finally able to get it to work by removing the audience from the VaultAuth object in kubernetes and of course the corresponding one in vault kubernetes auth configuration. If I put an invalid audience that doesn’t match the one in vault, then I get a different error that references the audience as the cause. But if I put the correct one, I get the above error.
I’m not an expert, but it seems like it may not be a vault error in accessing the secret as when I manually make the calls to vault using the vault command line or curl to the API everything works with the tokens created by kubernetes. But perhaps the TokenReview API call from vault is not including the audience claim? I’m not too familiar with that process to troubleshoot much further.
My question, is there a workaround or configuration that would allow use of audiences in the short lived service token flow, or is this not supported with an external vault installation?
This is a simplified version values I’m using in the helm configuration for VSO to create a default connection and auth:
defaultVaultConnection:
enabled: true
address: "https://mydomain:8200"
skipTLSVerify: false
defaultAuthMethod:
enabled: true
allowedNamespaces: ["*"]
method: kubernetes
mount: kubernetes-auth
kubernetes:
role: vso-operator
serviceAccount: vso-account
tokenAudiences: ["vault"]
And this is a simplified version of the configuration I’m applying to kubernetes to set up the service account, cluster role binding and vault static secret in a various namespaces reusing the default connection and auth, though I also tried in the vault-secrets-operator-system namespace.
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: test
name: vso-account
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vso-account
namespace: test
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
namespace: test
name: vso-auth-secret
spec:
mount: kv-k8s
type: kv-v2
path: test
refreshAfter: 60s
namespace: test
destination:
create: true
name: test-password
Not sure what else is relevant, but the kubernetes auth method in vault is configured with the “Disable use of local CA and service account JWT” enabled and kubernetes is using a CA that is trusted by the vault server. The vault server is using a let’s encrypt certificate for TLS generated by the router. And the vault-operator role in vault is set up with * as the bound service account and account namespace to rule those out as issues.
