We recently upgraded to Vault 1.10.3 on our development clusters. We make use of the vault agent injector to inject secrets into other services. After the upgrade we’re seeing the following errors both in the vault sidecar as well as the vault container itself.
Vault Sidecar Logs:
URL: PUT http://vault.default.svc:8200/v1/sys/leases/renew
Code: 400. Errors:
* lease not found
2022-10-17T07:05:38.818Z [WARN] vault.read(aws/creds/asgard): renewer done (maybe the lease expired)
2022-10-17T07:05:40.047Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/aws"
2022-10-17T07:11:08.991Z [WARN] vault.read(rabbitmq/creds/asgard): failed to renew: Error making API request.
URL: PUT http://vault.default.svc:8200/v1/sys/leases/renew
Code: 400. Errors:
* lease not found
2022-10-17T07:11:08.991Z [WARN] vault.read(rabbitmq/creds/asgard): renewer done (maybe the lease expired)
2022-10-17T07:11:09.295Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/rmq"
2022-10-17T07:20:09.414Z [WARN] vault.read(aws/creds/asgard): renewer done (maybe the lease expired)
2022-10-17T07:20:10.645Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/aws"
Vault Service Logs:
username: kubernetes-default-asgard-1d4815f4-e782-6a33-f35d-13de8b466956
2022-10-17T06:50:26.667Z [INFO] expiration: revoked lease: lease_id=aws/creds/asgard/iYs6Gx71sP8fqvgvtAMNteYj
2022-10-17T06:51:21.535Z [ERROR] secrets.system.system_cf93c3cc: lease renewal failed: lease_id=aws/creds/asgard/IqDTdxUldaE1NgC3XsX9hEDm error="lease not found"
2022-10-17T06:51:51.117Z [INFO] expiration: revoked lease: lease_id=rabbitmq/creds/asgard/PTq1SPE54FoshLKKbrSAaWn0
2022-10-17T06:53:18.519Z [INFO] expiration: revoked lease: lease_id=rabbitmq/creds/asgard/ou8SHHKWI2Tay54VXhhHbake
2022-10-17T06:53:53.981Z [ERROR] secrets.system.system_cf93c3cc: lease renewal failed: lease_id=aws/creds/asgard/iYs6Gx71sP8fqvgvtAMNteYj error="lease not found"
2022-10-17T06:54:52.531Z [ERROR] secrets.system.system_cf93c3cc: lease renewal failed: lease_id=rabbitmq/creds/janus/9vGtMMCvcUyie6FW4Db7UKcb error="lease not found"
username: kubernetes-default-janus-01469ebf-0496-ea7d-fc8b-54b477b9aa42
username: kubernetes-default-janus-119fe492-73b1-4f80-64ea-96d4cb450dcd
2022-10-17T06:55:41.250Z [TRACE] activity: writing segment on timer expiration
2022-10-17T06:55:41.449Z [DEBUG] core.autoseal: seal health test passed
2022-10-17T06:59:14.654Z [ERROR] secrets.system.system_cf93c3cc: lease renewal failed: lease_id=rabbitmq/creds/asgard/PTq1SPE54FoshLKKbrSAaWn0 error="lease not found"
username: kubernetes-default-asgard-696e50c9-115c-d4de-c4c6-58a9dc4e7715
2022-10-17T07:02:16.784Z [INFO] expiration: revoked lease: lease_id=aws/creds/asgard/YNQGD7zr28eH7EvrHh0LCQXB
2022-10-17T07:05:10.235Z [INFO] expiration: revoked lease: lease_id=rabbitmq/creds/asgard/Uy7YVbGP7yX5mhve7lW1CFuf
2022-10-17T07:05:41.250Z [TRACE] activity: writing segment on timer expiration
2022-10-17T07:05:41.464Z [DEBUG] core.autoseal: seal health test passed
2022-10-17T07:05:45.430Z [INFO] expiration: revoked lease: lease_id=aws/creds/asgard/hjGzHoP5ThdBMho4kIEh560N
2022-10-17T07:09:10.907Z [ERROR] secrets.system.system_cf93c3cc: lease renewal failed: lease_id=aws/creds/asgard/hjGzHoP5ThdBMho4kIEh560N error="lease not found"
2022-10-17T07:11:33.210Z [INFO] expiration: revoked lease: lease_id=rabbitmq/creds/janus/r8sM9LwvAPCz2hpVYsZRLwUe
2022-10-17T07:12:41.570Z [ERROR] secrets.system.system_cf93c3cc: lease renewal failed: lease_id=rabbitmq/creds/asgard/Tf7U1tlJuvKdRYZ8wDK2z3kP error="lease not found"
username: kubernetes-default-asgard-9ae61f97-37e6-6cfb-5ce7-7dfd3db2db09
2022-10-17T07:15:41.250Z [TRACE] activity: writing segment on timer expiration
2022-10-17T07:15:41.452Z [DEBUG] core.autoseal: seal health test passed
2022-10-17T07:15:55.505Z [INFO] expiration: revoked lease: lease_id=rabbitmq/creds/asgard/Vgvv0tAxpu7Mr5mfSI858Fw8
2022-10-17T07:17:33.693Z [INFO] expiration: revoked lease: lease_id=aws/creds/asgard/4N8dQTmKDYMTwCA4x0tpZ8Bu
Haven’t been able to figure out what’s the reason for this.