Is there any guide to troubleshoot k8s auth method?

Vault
1

I setup Kubernetes auth in vault and installed vault secret operator, and trying to make it working as described in The Vault Secrets Operator on Kubernetes | Vault | HashiCorp Developer but failed, the VaultStaticSecret created keep complaining about 403 error

  Warning  VaultClientConfigError  6m40s (x18 over 17m)  VaultStaticSecret  Failed to get Vault auth login: Error making API request.

URL: PUT https://keycloak2.intra.douban.com:8200/v1/auth/borin/login
Code: 403. Errors:

I’m trying to troubleshoot this but it’s hard.

I’d like to start with the k8s auth method, checking if I can pass the authentication using

vault  write -address=https://my-vault:8200 auth/borin/login role=borin-vso jwt=xxx

# it returns :
Error writing data to auth/borin/login: Error making API request.

URL: PUT https://my-vault:8200/v1/auth/borin/login
Code: 403. Errors:

* permission denied

where the JWT token was created with command

kubectl -n vault create token default --duration=3600s

it can be decoded as

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1702465731,
  "iat": 1702462131,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "vault",
    "serviceaccount": {
      "name": "default",
      "uid": "68a01ee6-f972-4527-8ec6-c2612703c2e3"
    }
  },
  "nbf": 1702462131,
  "sub": "system:serviceaccount:vault:default"
}

So what is the correct way to check my k8s auth setup?

How can I finally make the vault secret operator working?

2

Trouble shooting is hard, for some extra “hints” enable the vault-audit logs, while it is still cryptic - most values are HMAC encoded - at least you will see some of the transactions:

Using file and stdout since they are running in containers, they will end up wherever your k8s logs go.

You can also increase the vault log level in config - some errors end up in the system logs.

Additionaly, since vault is looping back to the k8s API to validate the JWT, there should be some errors in Kubernetes - though that’s not my expertise.

In the end, the understanding of what is going on is key.

Since you are not following the demo to the letter ( I see your auth path is borin ), It may help to how you set up the endpoint, link and role. What is your version of this:

vault write auth/demo-auth-mount/role/role1 \
   bound_service_account_names=default \
   bound_service_account_namespaces=app \
   policies=dev \
   audience=vault \
   ttl=24h

just to be certain it matches your JWT information.

3

@alain really appreciate your quick reply, I just enabled debug log with

log_requests_level = "debug"
log_level = "debug"

in vault config file

then I saw the real error:

2023-12-13 19:17:18.165864500 2023-12-13T19:17:18.165+0800 [DEBUG] auth.kubernetes.auth_kubernetes_db814b07: login unauthorized: err="Post \"https://sa-borin.k8s.xxx:443/apis/authentication.k8s.io/v1/tokenreviews\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

it turns out that I misconfigured the roor ca for api server, things are going right now.

Thanks for your help!

4

@LeoQuote could you help understand the steps you taken to resolve this issue ?

5

Refer to the vault manual for k8s auth Kubernetes - Auth Methods | Vault | HashiCorp Developer and reconfigure it with correct root ca.

You can submit another issue if you have any questions, as you can see, they respond quite quickly.