Vault helm deployment print out credentials in kubectl logs

Hi,

We have been following GitHub - hashicorp/vault-helm: Helm chart to install Vault and other associated components. for deployment.
We have found that stdout will always include the decrypted credential in return, meaning anyone who can run “kubectl logs” can see these credentials.
We have tried to put the log level to warn, but still it prints every single credential.

Is it because we are using a bit old version of the chart as well as the app?
vault-0.13.0 1.7.3

Thanks!

0.13.0 is quite old so it’s possible. I’ve been playing around with 0.19 and do not see any credentials leaks during deployment.

Keep in mind that Hashicorp recommends that you do not run vault in an open environment and to keep it completely sanctioned off in it’s own space – so really nobody should have access to your vault logs other than the vault/kub admins.

Thanks for the info, we will try to upgrade.
No, it is a closed env, but still not comfortable to see that kind of print out…

I pulled down 0.13 and I don’t see any secrets being exposed. What are you seeing exactly?

We are primarily using the credentials to serve the CICD pipelines in Jenkins (via vault plugin, and use approle to connect to vault). I see every response of the request is included in the pod log, and that includes the credentials being asked, if I run "kubectl logs vault-1 for example).
Sorry for not being clear about it. I haven’t paid attention if any leaks when we click the GUI, I can check that and come back.

In the beginning I thought it is due to the log level, as we initially did not modify the default logging level in server section. But surprisingly it is the same even if we change this to warn.

We haven’t tried to uplift or experient the new versions, I can come back if we see any differences in newer versions.

The Jenkins plugin is written pretty openly and have a lot of security issues. It’s opensource plugin so if you can contribute back to it and fix the issues that would be ideal.

Not sure if I haven’t made it clear, the print out of credentials are in the logs of vault, not in Jenkins.
Jenkins just used some api to get credential I suppose, and I guess this is anyhow a vault issue to print out the response including credential in its logs.

Sorry I must have misunderstood. I have deployed a couple of different versions of the helm chart both at work and at home and have not seen any leakage. Can you share an example of this?

Hi,

I finally know why now…
It is nothing to do with any log level or deployment.

Someone enabled the audit log… without disabling it.
I am sorry for wasting your time, I also tried everything to reproduce, and not possible as you say.
Vault was installed by another guy who has left the job, and we are still in the very beginning phase of the work.

Ah ok. I should mention that Audit logs are “sensitive” information and if you have a large enough installation become an necessary evil, but should be protected. They don’t actually contain the actual secrets though.

They do, at least they return response from approle for that version. But at least now we have disabled that and it is a closed env, thanks a lot again!

By default, the response data is hashed, but it sounds like a previous admin may have made some risky choices in the audit_non_hmac_response_keys setting for your approle mount.

To inspect the settings:

$ vault read sys/auth/approle/tune
Key                             Value
---                             -----
audit_non_hmac_response_keys    [foobar]
default_lease_ttl               720h
description                     n/a
force_no_cache                  false
max_lease_ttl                   1440h
token_type                      default-service

To update them:

$ vault auth tune -audit-non-hmac-response-keys foobar approle
Success! Tuned the auth method at: approle/

The audit log can be extremely useful for diagnosing application issues, so consider reviewing all these settings, so you can confidently enable it again!

It is possible to do some manipulation of the audit/auth devices, check to make sure they didn’t do anything non-standard.

If it persist, I’d highly recommend deleting the approle and recreating it. It would rotate the appid – so only do that if it isn’t a big deal to replace it in whatever app(s) use it.