Ember Data Request GET /v1/sys/policies/acl returned a 500 Payload (application/json)

Vault
1

Hello all, first post to the group :slight_smile:

I inherited a vault 1.6.1 + consul cluster and Im experiencing some issues with 2 separate areas.

When i try to view the polices - i get the following error in the UI
Ember Data Request GET /v1/sys/policies/acl returned a 500 Payload (application/json) [object Object]

I get the similar error when trying to access one of our secrets
Ember Data Request GET /v1/secret/config/ returned a 500 Payload (application/json) [object Object]

When accessing via the CLI i get the same 500 error trying to list either of these 2 sections

however, I CAN access the secrets within the /secret/config/ via the UI and CLI if i specify the absolute path to the secret, as well as access other secrets in locations (i.e. secret/data secret/backup and so on)

I cannot access the policies this way, its either a 500 error above, or a 404 not found.

All of our processes are still working, automated jobs can get secrets and so on, so technically things are not “broken”. Its just the 500 errors

Any suggestions on where to start poking around?

Thanks in advance

Chris

2

I’d look at increasing the verbosity of the logging of the Vault service to see if you’re hitting any backend errors while hitting those endpoints.

3

What @jeffsanicola said, and also are you using Curl against Vault? Or you trying to pull the policies from Consul? They’re not the same thing.

4

Thanks Jeff - we will be turning up logging on the next maintenance window - I will post anything we learn from the logs

as for the commands:
Listing the secret/config fails:

curl -H “X-Vault-Token: XXXXXXXXXXX” -X LIST https://vault-1myhost.com:8200/v1/secret/config
{“errors”:[“internal error”]}

but accessing the contents like this works:

curl -H “X-Vault-Token: XXXXXXXXXXX” -X GET https://vault-1myhost.com:8200/v1/secret/config/my-secret

Direct CLI on the server yeilds the same results:

vault list secret/config/
Error listing secret/config/: Error making API request.

URL: GET https://vault-1.myhost.com:8200/v1/secret/config?list=true
Code: 500. Errors:

  • internal error

but reading its contents works

vault read secret/config/my-secret
Key Value

refresh_interval 8760h

the UI exhibits the same behavior as well,

if i click on “config” under secret i get the error in the subject line

Ember Data Request GET /v1/secret/config/ returned a 500 Payload (application/json) [object Object] internal error

but if i type the path directly to the my-secret it works fine.

All commands are executed against vault, all data is stored in consul.

Thanks for the guidance

Chris

5

What are you trying to do (and don’t say “run these commands”) What is the purpose of your commands.

You can’t LIST a config, it’s a map.

This isn’t valid either. Assuming this is a KV2, you’re missing the data path.

Again, can’t list a config.

$ curl -s -X GET -H "X-Vault-Token: $(vault print token)" https://vault/v1/secret/config | jq .{
  "request_id": "c7b8067a-3840-debd-b1b0-8a5248011e79",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "cas_required": false,
    "delete_version_after": "0s",
    "max_versions": 0
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}
6

thanks for the reply, in our environment - all of the commands I listed above work in our qa/pre-prod/stageing/production -dr, and production environments but in DEV (the systems im seeing the errors) they do NOT work

here are some example running the cli and curl commands above on different environments in our Datacenter

Production DR:

CLI:
vault-2:~$ vault list secret/config
Keys

api-gateway,production-dr
appointment-service,production-dr
billing-adapter,production-dr
billing-event,production-dr

(there are over 200 more entries i didn’t list but you get the idea)

and if I curl against that environment

CURL:

curl -H “X-Vault-Token: XXXXXXXXXXXXXXXXXXX” -X LIST https://vault-1.my-production-dr.com:8200/v1/secret/config
{“request_id”:“7186164c-3958-af44-a0d6-f4fe173801b2”,“lease_id”:"",“renewable”:false,“lease_duration”:0,“data”:{“keys”:[“api-gateway,production-dr”,“appointment-service,production-dr”,“billing-adapter,production-dr”,“billing-event,production-dr”,

again several pages of output

The reason for this troubleshooting is that users were able to add K/V items to the secret/config from the UI until a few days ago - there are about 290 entries in this secret currently - all manually entered through the UI over the past few years, then this error started happening and they passed it off to me :slight_smile:

again all command (cli and curl) i posted above work in 5 environments - but NOT in the 6th - The only difference in all of the environments is process - they are all physically identical accept developers have root access to the dev cluster - and NO access to the others

So my thought is that someone “broke it” i don’t know how (why im here) just trying to fix it

7

Update - this is also happening on the sys/policy - same error, same situation
In the UI if I go to the top menu and click on Policies - i get the same 500 error, BUT if i know that PATH to the policy - i can view and edit in the UI.

on the cli and curl
vault-1:~$ vault policy list
Error listing policies: Error making API request.

URL: GET https://vault-1.dev.myvault.com:8200/v1/sys/policies/acl?list=true
Code: 500. Errors:

  • internal error

curl -H “X-Vault-Token: xxxxxxx” -X LIST https://vault-1.dev.myvault.com:8200/v1/sys/policy/
{“errors”:[“internal error”]}

but direct requests to policies work
curl -H “X-Vault-Token: xxxxxxxxxxxxxxxxxx” -X GET https://vault-1.dev.myvault.com:8200/v1/sys/policy/root

{“name”:“root”,“rules”:"",“request_id”:“59bee75b-a9ce-2a53-3101-68eaf4ad563c”,“lease_id”:"",“renewable”:false,“lease_duration”:0,“data”:{“name”:“root”,“rules”:""},“wrap_info”:null,“warnings”:null,“auth”:null}

8

Turn on DEBUG logging on the vault configuration. Generate the error and see what it shows or points at.

9

Hello,
I have a similar behavior that is easy to reproduce.
If more than 200 secret engine mount points are created in the same vault namespace, it is no longer possible to display the secret engine list. The UI and the vault secrets list command return an error 500.
Secrets can still be accessed directly, but it is no longer possible to list them.
However, on the Limits and Maximums | Vault | HashiCorp Developer page it is indicated that it is possible to have 7000 secret engine mount points with Consul and 14000 with integrated storage default.

10

The bug I encountered is described in the issue "event not processed by enough 'sink' nodes" affecting audit requests · Issue #24376 · hashicorp/vault · GitHub
Apparently the patch in vault version 1.15.5 doesn’t fix the problem. To display all entries, you still need to disable auditing.