# Debugging vault policies

**URL:** <https://discuss.hashicorp.com/t/debugging-vault-policies/53785>\
**Category:** Vault\
**Created:** [May 15, 2023, 10:10am UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785 "2023-05-15T10:10:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![andybusuu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andybusuu/32/35699_2.png) [@andybusuu](https://discuss.hashicorp.com/u/andybusuu)\
**Post date:** [May 15, 2023, 10:10am UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/1 "2023-05-15T10:10:57Z")

</div>

We are using GitHub teams to allocate policies to users.

On logging in via the vault cli I can see the policies associated with my user. On the whole these look correct, however I also see one application specific policy `foo-app`. How can I debug why that application specific policy is associated with my user?

N.B. It’s possible that in the past that the `foo-app` policy was associated with my user account, but the vault policies have since been updated. Does vault “remember” these historical associations?

My GitHub team policy is `admin`

```auto
- type: github
        config:
          organization: Busuu
        map:
          teams:
            AWS: default,webui-helper,admin

```

The `admin` policy:

```auto
     - name: admin
        rules: |
          path "sys/*" {
            capabilities = ["list", "read", "delete"]
          }
          path "static/*" {
            capabilities = ["list", "create", "read", "update", "delete"]
          }
          path "static/metadata" {
            capabilities = ["list"]
          }
          path "sys/internal/ui/mounts/static" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "static/data/*" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "database/creds/*" {
            capabilities = ["read"]
          }
          # These are to allow for manual OIDC setup
          path "sys/auth/oidc" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "auth/oidc/*" {
            capabilities = ["list", "create", "read", "update", "delete"]
          }

          # Full access to policies
          # List existing policies
          path "sys/policies/acl"
          {
            capabilities = ["list"]
          }
          # Create and manage ACL policies
          path "sys/policies/acl/*"
          {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          # Enable and manage authentication methods broadly across Vault
          # Manage auth methods broadly across Vault
          path "auth/*"
          {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          # Create, update, and delete auth methods
          path "sys/auth/*"
          {
            capabilities = ["create", "update", "delete", "sudo"]
          }
          # List auth methods
          path "sys/auth"
          {
            capabilities = ["read"]
          }

```

The `foo-app` policy:

```auto
      - name: foo-app
        rules: |
          path "static/data/ateam/foo-app" {
            capabilities = ["list", "read"]
          }

```

The problem is I’m able to edit secrets under:  
`ateam/bar-app/`

But I only have read-access to:  
`ateam/foo-app/`

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [May 15, 2023, 4:38pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/2 "2023-05-15T16:38:59Z")

</div>

> [@andybusuu](#):
>
> How can I debug why that application specific policy is associated with my user?

There is no “explain” functionality in Vault for this so it requires manually checking information in various places.

The starting point is to use `vault token lookup` - please do that and paste the output, however take care to omit the line starting with `id` which is **your actual secret token**!

The main thing we’re looking for here is to figure out whether the policy is being attached via the `identity_policies` or `policies` list.

If it’s `identity_policies`, then the policy must be linked to your identity entity, or an identity group that it is a member of. You would then need to explore the identity secrets engine data via its API to track down which.

If it’s `policies`, then it was associated with the token at the time of creation, by configuration related to the auth method. Exact APIs to explore would depend on the specific auth method in use.

* * *

Apart from that, there’s a lot I’d like to say about your `admin` policy’s contents. It’s not exactly broken, but it does contain a lot of things which are conspicuously redundant to an expert.

> [@andybusuu](#):
>
> ```auto
> path "static/metadata" {
> capabilities = ["list"]
> }
> 
> ```

This is practically redundant with the rule for `static/*`.

> [@andybusuu](#):
>
> ```auto
> path "sys/internal/ui/mounts/static" {
> capabilities = ["create", "read", "update", "delete", "sudo"]
> }
> 
> ```

You should not have any policy statements for `sys/internal/ui/mounts/...` - this is a special endpoint which bypasses the policy system anyway, and does its own access control based on which secret engines do have any access granted to them.

> [@andybusuu](#):
>
> ```auto
> path "static/data/*" {
> capabilities = ["create", "read", "update", "delete", "sudo"]
> }
> 
> ```

This is functionally redundant with the wider `static/*` rule, given that neither the `list` nor `sudo` capabilities have any meaning for `data/*` in a KV v2 secrets engine.

> [@andybusuu](#):
>
> ```auto
> path "sys/policies/acl"
> {
> capabilities = ["list"]
> }
> 
> ```

This is redundant with the `sys/policies/acl/*` block that appears next. (And the mention of `sudo` in that block is also redundant, as the `sys/policies/acl/*` APIs do not make use of `sudo` checks.)

---

<div class="post-metadata">

**Author:** ![andybusuu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andybusuu/32/35699_2.png) [@andybusuu](https://discuss.hashicorp.com/u/andybusuu)\
**Post date:** [May 24, 2023, 1:48pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/3 "2023-05-24T13:48:48Z")

</div>

> [@maxb](#):
>
> The main thing we’re looking for here is to figure out whether the policy is being attached via the `identity_policies` or `policies` list.

The `vault token lookup` command returns this for me:

```auto
Error looking up token: Error making API request.

URL: GET https://vault.foocorp.net/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

```

However when running `vault login -method=github token="$VAULT_TOKEN"` I see:

```auto
WARNING! The VAULT_TOKEN environment variable is set! The value of this
variable will take precedence; if this is unwanted please unset VAULT_TOKEN or
update its value accordingly.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token ****
token_accessor ****
token_duration 768h
token_renewable true
token_policies ["admin" "b2b-secrets" "backend-secrets" "default" "webui-helper"]
identity_policies []
policies ["admin" "b2b-secrets" "backend-secrets" "default" "webui-helper"]
token_meta_org FooCorp
token_meta_username andybusuu

```

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [May 24, 2023, 1:59pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/4 "2023-05-24T13:59:41Z")

</div>

> [@andybusuu](#):
>
> ```auto
> token ****
> 
> ```

You’d best revoke this ASAP, or anyone at your organization could use it to access your Vault as you!

* * *

The fact that you get permission denied when running `vault token lookup` tells me that you’ve done something unusual and undesirable to your Vault installation. Permission to `read` the path `auth/token/lookup-self` is provided in the default contents of the `default` policy - it seems either you’ve changed that to be less permissive, or you’ve accidentally overridden that fairly basic access in another policy.

* * *

The fact that you have a token starting with `s.` instead of `hvs.` suggests you are running a rather old version of Vault, and really should upgrade soon.

* * *

The fact that all your policies are being assigned as `token_policies` tells me they are all coming from definitions within the GitHub auth method in Vault.

You need to next look at:

```auto
vault read auth/github/config
vault list auth/github/map/teams/
vault read auth/github/map/teams/<each team name listed>
vault list auth/github/map/users/
vault read auth/github/map/users/<each user name listed>

```

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [May 24, 2023, 2:00pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/5 "2023-05-24T14:00:49Z")

</div>

Although you have now edited the token value out of your previous message, please be aware that the forum software still allows users to browse older versions of edited comments.

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [May 24, 2023, 2:07pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/6 "2023-05-24T14:07:48Z")

</div>

> [@andybusuu](#):
>
> However when running `vault login -method=github token="$VAULT_TOKEN"` I see:

Actually, now I look more closely at what exactly you ran there, that command is very wrong.

The `VAULT_TOKEN` environment variable is supposed to contain a **Vault token**.

Yet, by the way you use it in that command, it apparently contains a **GitHub token**.

This will break all future Vault CLI commands you attempt. That’s what the warning printed by the CLI:

> [@andybusuu](#):
>
> ```auto
> WARNING! The VAULT_TOKEN environment variable is set! The value of this
> variable will take precedence; if this is unwanted please unset VAULT_TOKEN or
> update its value accordingly.
> 
> ```

is trying to tell you.

---

<div class="post-metadata">

**Author:** ![andybusuu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andybusuu/32/35699_2.png) [@andybusuu](https://discuss.hashicorp.com/u/andybusuu)\
**Post date:** [May 24, 2023, 4:21pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/7 "2023-05-24T16:21:51Z")

</div>

Thanks! I updated `VAULT_TOKEN` with the value from `vault login` and now all requests are working.

`export VAULT_TOKEN=" s.sOvN ****"`

N.B. We are using the BanzaiCloud bank vaults helm chart to install vault

```auto
helmCharts:
- name: vault
  version: 1.15.3
  repo: https://kubernetes-charts.banzaicloud.com

```

This provisions vault `Hashicorp Vault 1.9.6`.

We define the “GitHub team” → “Vault policy” mappings in a values.yaml file. On running this command I noticed there is an out of date team still defined even though they have since been removed from the values.yaml.

Crucially I see that it is this team that is giving my user the policies that I don’t want.

What is the best way to remove that state team mapping?

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [May 24, 2023, 7:02pm UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/8 "2023-05-24T19:02:52Z")

</div>

> [@andybusuu](#):
>
> This provisions vault `Hashicorp Vault 1.9.6`.

This version is out of support. HashiCorp operate a “three most recent branches” policy, meaning currently 1.11, 1.12, 1.13.

You should review how you implement moving to newer Vault release branches.

* * *

> [@andybusuu](#):
>
> On running this command I noticed there is an out of date team still defined even though they have since been removed from the values.yaml.

It is very easy to implement some naive basic automation that copies entries from a YAML file to the Vault API, and forget to deal with cleaning up mappings that are removed from the YAML file entirely.

My guess is that’s what happen here.

Long term, you’d need to fix the automation to reliably clean up after itself.

Short term, you might opt for an ad-hoc fix, such as manually deleting the extra mapping from the Vault API, or adding the out of date team back in to the automation, associated with an empty list of policies.

---

<div class="post-metadata">

**Author:** ![andybusuu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andybusuu/32/35699_2.png) [@andybusuu](https://discuss.hashicorp.com/u/andybusuu)\
**Post date:** [May 25, 2023, 11:16am UTC](https://discuss.hashicorp.com/t/debugging-vault-policies/53785/9 "2023-05-25T11:16:22Z")

</div>

Hi, I deleted the stale team mappings and now policies are behaving how I expect.

Thanks so much for the assistance! 🙏

We will look into upgrading our Vault version. 👍