We have configured our vault instance to support the consul engine for provisioning time limited consul tokens. These are used by both humans and services. We set the TTL for 30 days and rotate the consul tokens as appropriate using scheduled jobs.
We have seen that at times these consul tokens will go missing from consul. i have to assume that vault is revoking these tokens however, there seems to be no trace in any log or audit event, so its very hard to troubleshoot. It results in these services and agent logs spewing errors for ACL not found.
Has anyone experienced this? We are at a point where we may need to ditch this approach for services and use either long lasting tokens or roll our own rotation mechanism.