Debugging vault policies

We are using GitHub teams to allocate policies to users.

On logging in via the vault cli I can see the policies associated with my user. On the whole these look correct, however I also see one application specific policy foo-app. How can I debug why that application specific policy is associated with my user?

N.B. It’s possible that in the past that the foo-app policy was associated with my user account, but the vault policies have since been updated. Does vault “remember” these historical associations?

My GitHub team policy is admin

- type: github
        config:
          organization: Busuu
        map:
          teams:
            AWS: default,webui-helper,admin

The admin policy:

     - name: admin
        rules: |
          path "sys/*" {
            capabilities = ["list", "read", "delete"]
          }
          path "static/*" {
            capabilities = ["list", "create", "read", "update", "delete"]
          }
          path "static/metadata" {
            capabilities = ["list"]
          }
          path "sys/internal/ui/mounts/static" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "static/data/*" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "database/creds/*" {
            capabilities = ["read"]
          }
          # These are to allow for manual OIDC setup
          path "sys/auth/oidc" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "auth/oidc/*" {
            capabilities = ["list", "create", "read", "update", "delete"]
          }

          # Full access to policies
          # List existing policies
          path "sys/policies/acl"
          {
            capabilities = ["list"]
          }
          # Create and manage ACL policies
          path "sys/policies/acl/*"
          {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          # Enable and manage authentication methods broadly across Vault
          # Manage auth methods broadly across Vault
          path "auth/*"
          {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          # Create, update, and delete auth methods
          path "sys/auth/*"
          {
            capabilities = ["create", "update", "delete", "sudo"]
          }
          # List auth methods
          path "sys/auth"
          {
            capabilities = ["read"]
          }

The foo-app policy:

      - name: foo-app
        rules: |
          path "static/data/ateam/foo-app" {
            capabilities = ["list", "read"]
          }

The problem is I’m able to edit secrets under:
ateam/bar-app/

But I only have read-access to:
ateam/foo-app/

There is no “explain” functionality in Vault for this so it requires manually checking information in various places.

The starting point is to use vault token lookup - please do that and paste the output, however take care to omit the line starting with id which is your actual secret token!

The main thing we’re looking for here is to figure out whether the policy is being attached via the identity_policies or policies list.

If it’s identity_policies, then the policy must be linked to your identity entity, or an identity group that it is a member of. You would then need to explore the identity secrets engine data via its API to track down which.

If it’s policies, then it was associated with the token at the time of creation, by configuration related to the auth method. Exact APIs to explore would depend on the specific auth method in use.


Apart from that, there’s a lot I’d like to say about your admin policy’s contents. It’s not exactly broken, but it does contain a lot of things which are conspicuously redundant to an expert.

This is practically redundant with the rule for static/*.

You should not have any policy statements for sys/internal/ui/mounts/... - this is a special endpoint which bypasses the policy system anyway, and does its own access control based on which secret engines do have any access granted to them.

This is functionally redundant with the wider static/* rule, given that neither the list nor sudo capabilities have any meaning for data/* in a KV v2 secrets engine.

This is redundant with the sys/policies/acl/* block that appears next. (And the mention of sudo in that block is also redundant, as the sys/policies/acl/* APIs do not make use of sudo checks.)

1 Like

The vault token lookup command returns this for me:

Error looking up token: Error making API request.

URL: GET https://vault.foocorp.net/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

However when running vault login -method=github token="$VAULT_TOKEN" I see:

WARNING! The VAULT_TOKEN environment variable is set! The value of this
variable will take precedence; if this is unwanted please unset VAULT_TOKEN or
update its value accordingly.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  ****
token_accessor         ****
token_duration         768h
token_renewable        true
token_policies         ["admin" "b2b-secrets" "backend-secrets" "default" "webui-helper"]
identity_policies      []
policies               ["admin" "b2b-secrets" "backend-secrets" "default" "webui-helper"]
token_meta_org         FooCorp
token_meta_username    andybusuu

You’d best revoke this ASAP, or anyone at your organization could use it to access your Vault as you!


The fact that you get permission denied when running vault token lookup tells me that you’ve done something unusual and undesirable to your Vault installation. Permission to read the path auth/token/lookup-self is provided in the default contents of the default policy - it seems either you’ve changed that to be less permissive, or you’ve accidentally overridden that fairly basic access in another policy.


The fact that you have a token starting with s. instead of hvs. suggests you are running a rather old version of Vault, and really should upgrade soon.


The fact that all your policies are being assigned as token_policies tells me they are all coming from definitions within the GitHub auth method in Vault.

You need to next look at:

vault read auth/github/config
vault list auth/github/map/teams/
vault read auth/github/map/teams/<each team name listed>
vault list auth/github/map/users/
vault read auth/github/map/users/<each user name listed>

Although you have now edited the token value out of your previous message, please be aware that the forum software still allows users to browse older versions of edited comments.

Actually, now I look more closely at what exactly you ran there, that command is very wrong.

The VAULT_TOKEN environment variable is supposed to contain a Vault token.

Yet, by the way you use it in that command, it apparently contains a GitHub token.

This will break all future Vault CLI commands you attempt. That’s what the warning printed by the CLI:

is trying to tell you.

Thanks! I updated VAULT_TOKEN with the value from vault login and now all requests are working.

export VAULT_TOKEN=" s.sOvN****"

N.B. We are using the BanzaiCloud bank vaults helm chart to install vault

helmCharts:
- name: vault
  version: 1.15.3
  repo: https://kubernetes-charts.banzaicloud.com

This provisions vault Hashicorp Vault 1.9.6.

We define the “GitHub team” → “Vault policy” mappings in a values.yaml file. On running this command I noticed there is an out of date team still defined even though they have since been removed from the values.yaml.

Crucially I see that it is this team that is giving my user the policies that I don’t want.

What is the best way to remove that state team mapping?

This version is out of support. HashiCorp operate a “three most recent branches” policy, meaning currently 1.11, 1.12, 1.13.

You should review how you implement moving to newer Vault release branches.


It is very easy to implement some naive basic automation that copies entries from a YAML file to the Vault API, and forget to deal with cleaning up mappings that are removed from the YAML file entirely.

My guess is that’s what happen here.

Long term, you’d need to fix the automation to reliably clean up after itself.

Short term, you might opt for an ad-hoc fix, such as manually deleting the extra mapping from the Vault API, or adding the out of date team back in to the automation, associated with an empty list of policies.

Hi, I deleted the stale team mappings and now policies are behaving how I expect.

Thanks so much for the assistance! :pray:

We will look into upgrading our Vault version. :+1: