Vault token lease time decrease on renewal

Nomad
#1

There are three Nomad servers in a cluster. They can obtain (using approle auth method) and renew authentication tokens from a HashiCorp Vault instance. But the token lease time decreases on each renew.

May 19 19:57:44 nomad-1 nomad[3053860]:     2022-05-19T19:57:36.084+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=383h59m59.499995415s
Jun 04 19:57:35 nomad-1 nomad[3053860]:     2022-06-04T19:57:35.716+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=191h59m59.99999415s
Jun 12 19:57:35 nomad-1 nomad[3053860]:     2022-06-12T19:57:35.908+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=95h59m59.99999487s
Jun 16 19:57:36 nomad-1 nomad[3053860]:     2022-06-16T19:57:36.038+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=47h59m59.999995055s
Jun 18 19:57:36 nomad-1 nomad[3053860]:     2022-06-18T19:57:36.184+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=23h59m59.499994375s
Jun 19 19:57:35 nomad-1 nomad[3053860]:     2022-06-19T19:57:35.820+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=11h59m59.999995195s
Jun 20 07:57:35 nomad-1 nomad[3053860]:     2022-06-20T07:57:35.958+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=5h59m59.99999519s
Jun 20 13:57:36 nomad-1 nomad[3053860]:     2022-06-20T13:57:36.093+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=2h59m59.499995635s
Jun 20 16:57:35 nomad-1 nomad[3053860]:     2022-06-20T16:57:35.730+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=1h29m59.99999516s
Jun 20 18:27:35 nomad-1 nomad[3053860]:     2022-06-20T18:27:35.870+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=44m59.99999546s
Jun 20 19:12:36 nomad-1 nomad[3053860]:     2022-06-20T19:12:36.005+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=22m29.999995125s
Jun 20 19:35:06 nomad-1 nomad[3053860]:     2022-06-20T19:35:06.163+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=11m14.49999277s
Jun 20 19:46:20 nomad-1 nomad[3053860]:     2022-06-20T19:46:20.803+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=5m37.499995675s
Jun 20 19:51:58 nomad-1 nomad[3053860]:     2022-06-20T19:51:58.448+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=2m48.49999554s
Jun 20 19:54:47 nomad-1 nomad[3053860]:     2022-06-20T19:54:47.081+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=1m23.999992335s
Jun 20 19:56:11 nomad-1 nomad[3053860]:     2022-06-20T19:56:11.197+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=41.99999536s
Jun 20 19:56:53 nomad-1 nomad[3053860]:     2022-06-20T19:56:53.311+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=20.999995325s
Jun 20 19:57:14 nomad-1 nomad[3053860]:     2022-06-20T19:57:14.425+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=10.49999523s
Jun 20 19:57:25 nomad-1 nomad[3053860]:     2022-06-20T19:57:25.041+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=5.49999391s
Jun 20 19:57:30 nomad-1 nomad[3053860]:     2022-06-20T19:57:30.657+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=2.49999536s
Jun 20 19:57:33 nomad-1 nomad[3053860]:     2022-06-20T19:57:33.275+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=999.995625ms
Jun 20 19:57:34 nomad-1 nomad[3053860]:     2022-06-20T19:57:34.391+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=499.9959ms
Jun 20 19:57:35 nomad-1 nomad[3053860]:     2022-06-20T19:57:35.007+0800 [INFO]  nomad.vault: successfully renewed token: next_renewal=499.99569ms
Jun 20 19:57:37 nomad-1 nomad[3053860]:     2022-06-20T19:57:37.108+0800 [WARN]  nomad.vault: got error or bad auth, so backing off:

Once lease time reaches zero no renewal is possible until the Nomad agent restart.
It doesn’t look like expected behavior. Please help me to figure out the problem.

#2

Hi @msheremet, can you confirm that you have a periodic token without a Max TTL set?

#3

Hi @seth.hoenig , I didn’t set explicit Max TTL. Not sure if there is any default value. Here is the output of vault token lookup command:

Key                  Value
---                  -----
accessor             tX4jIEf1eGF9Uwi9Fjoq6WsD
creation_time        1655913805
creation_ttl         768h
display_name         approle
entity_id            025477ac-c30d-b0b2-3c6e-e5469fdd1289
expire_time          2022-07-25T00:03:25.774977497+08:00
explicit_max_ttl     0s
id                   n/a
issue_time           2022-06-23T00:03:25.32588749+08:00
last_renewal         2022-06-23T00:03:25.774977717+08:00
last_renewal_time    1655913805
meta                 map[role_name:nomad-server]
num_uses             0
orphan               true
path                 auth/approle/login
policies             [default nomad-server]
renewable            true
ttl                  719h26m42s
type                 service