We have configured our vault instance to support the consul engine for provisioning time limited consul tokens. These are used by both humans and services. We set the TTL for 30 days and rotate the consul tokens as appropriate using scheduled jobs.
We have seen that at times these consul tokens will go missing from consul. i have to assume that vault is revoking these tokens however, there seems to be no trace in any log or audit event, so its very hard to troubleshoot. It results in these services and agent logs spewing errors for ACL not found.
Has anyone experienced this? We are at a point where we may need to ditch this approach for services and use either long lasting tokens or roll our own rotation mechanism.
A Vault token that expires, also removes all associate child tokens with it. I doubt that there is a way to keep them completely in sync, so that’s the most likely scenario. There should be an entry in the audit log that the token expired and was removed, most likely you’ll find the consul token disappeared at the same time, less a few milliseconds.
So, if the token used to make the api request to the consul mount point expires, vault will revoke all consul secret engine leases associated with it? How are you suppose to use short lived tokens if everything it generates gets revoked?
In vault the solution is -orphan the token at creation but I’m not sure if that’s doable in your situation. My consul usage is as a storage backend for Vault.