Trying to set up Boundary within k8s using Vault dynamic database credentials and while working on it I watched as Vault would renew the token and extend the PostgreSQL user credentials another hour as I had TTL of 1h and Max TTL of 24h set on the role. The issue seemed to arise after the max TTL of 24h was reached and new credentials were issued.
I quickly resolved the issue by restarting the controller pods which got new credentials generated but in doing so I lost the ability to actually dig around to troubleshoot and confirm. From the PostgreSQL logs, I could see at the 24h mark the ERROR: permission denied for table <table name> errors began and continued until I restarted the controller pods. This leads me to believe that the Boundary controller is not acknowledging the credentials in the secret changed and re-establish the connection with the new credentials and it while it hasn’t been disconnected it proceeds to try and use the credentials that originally allowed it to connect.
I fully expect to wake up and get online tomorrow and see things in the same state but will take more time to troubleshoot and confirm before restarting things so quickly. I don’t believe the issue is with Vault as it appeared to be doing everything correctly. The lease would be renewed up until the max TTL which didn’t change the username or the password values in the secret. What I suspect, but failed to verify by restarting the pods, was that the password and/or username changed at the max TTL (24h) mark and the Vault Agent sidecar updated the secret but Boundary didn’t reload and then re-establish the database connectivity.