Hi,
I’m having an odd problem where when using SaltStack with proxy minons, it’s not obtaining a new vault token on every run, but it keeps trying to use the same one - even after it’s expired.
Raised an issue here:
opened 02:52PM - 21 Jan 22 UTC
Bug
needs-triage
**Description**
We have two salt setups - one using regular minions to provis… ion servers, which we have been doing for over a year and another using proxy minions to provision our network devices, which we have recently started using. We use Hashicorp Vault for secrets.
I am using the same configuration and utilising vault in the same way between the two setups. It works fine on the server side with regular minions. However, on the network side with proxy minions, something strange is going on with regard to refreshing tokens.
Using `state.highstate` seems to work fine. However, if I use other functions like: `net.load_template` or `slsutil.renderer`, it will work fine for the TTL of the token, then i'll get an error like:
```
salt.exceptions.SaltRenderError: Problem running salt function in Jinja template: Failed to read secret! HTTPError: 403 Client Error: Forbidden for url: https://myhost/v1/path/to/secret; line 5
---
[snip]
{%- set headers = { 'X-API-Key': salt['vault'].read_secret('/path/to/secret').key } -%} <======================
```
It looks like it's using an old token which has now expired!
Strangely, if I do a `state.highstate`, that'll then work fine - but going back to `net.load_template` or `slsutil.renderer`, they'll still error. So... `state.highstate` appears to be doing the right thing - but strangely not triggering a token refresh (to buy another 5 mins) globally.
The only way I can find of fixing it for these other commands is to either wait (trying again later on and it might work again - as if the old token eventually expires and causes a refresh) or to restart the proxy minion. Then it'll work fine again for $TTL and break again.
**Setup**
We are running our own Salt masters, on Debian 11 (bullseye) VMs.
Using the latest salt versions from the Saltstack Debian repos.
```
root@network-master:~# dpkg -l |grep salt
ii salt-common 3004+ds-1 all shared libraries that salt requires for all packages
ii salt-master 3004+ds-1 all remote manager to administer servers via salt
ii salt-minion 3004+ds-1 all client package for salt, the distributed remote execution system
```
We are configuring the salt master with:
```
vault:
url: https://<URL>
auth:
method: token
token: <REDACTED>
uses: 30
ttl: 300
policies:
- something
- something2
peer_run:
.*:
- vault.generate_token
```
I then do things like this in jinja templates:
```
{{ salt['vault'].read_secret('/path/to/secret').password }}
```
**Expected behavior**
On the server/minion side, if I do:
```
root@salt-master:~# salt <minion> vault.read_secret auth/token/lookup-self
```
... on each run I get a different token - each with an expiry of +5 mins, as you'd expect.
**Steps to Reproduce the behavior**
On the network/proxy minion side, if I do:
```
root@network-master:~# salt <proxy minion> vault.read_secret auth/token/lookup-self
```
... on every run it's the same token - so it appears that instead of going to vault and obtaining a token on every run as it should, it's deciding to get one sometimes and re-using it...and continuing to try and use it even after it's expires... until some point later, when it decides to go and get another one.
**Versions Report**
```
root@network-master:~# salt --versions-report
Salt Version:
Salt: 3004
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: 1.1.0
M2Crypto: 0.37.1
Mako: Not Installed
msgpack: 1.0.0
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.9.7
pygit2: 1.4.0
Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 20.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: debian 11 bullseye
locale: utf-8
machine: x86_64
release: 5.10.0-9-amd64
system: Linux
version: Debian GNU/Linux 11 bullseye
```
</details>
**Additional context**
If I set a higher TTL and uses, then it makes it work for longer, as there is longer before the token expires.... but then it still happens when it does expire.
Anyone had a similar problem?
Thanks,
Ian