In short I have 2 questions. First would be - is it possible in overall with an automated script to copy aws dynamically generated IAM user credential keys and TTL and via script put them in to secret engine kv1 or kv2 with defined TTL to automate key rotation ?
And is it possible to create a sidecar that could do this individually for each application or it’s better if it’s a single app/pod transferring to aws dynamic secrets in to one of the secret engines (kv1;kv2) ?
The K/V secret engine doesn’t have TTLs/leases.
I’m not sure why you’d want to use the K/V engine at all? Just pull the IAM user details directly from your application, extending the lease/requesting new credentials as needed.
Basically the idea is to transfer IAM secret keys to one of K/V engine’s and use the TTL for the app to identify when to request the new pair of keys so in the end TTL doesn’t get put in to K/V secret engine, as you said it yourself it doesn’t support that.
After that transferred key can be injected in to application via banzaiclouds secret webhook, so using K/V secret seems to make sense to not expose the keys in plain text.
As well the env which “transfer” the keys to the K/V engine would be cleaned up so key’s are not recoverable either.
I’m still not understanding why your app can’t get the credentials directly from the AWS secret engine as is standard? The lease will have a TTL and can be either renewed by the application to extend their lifetime or request new credentials if the lease expires (e.g. due to max TTL).
Maybe it was my wording, I apologize. I will try to explain exactly what I want to find out.
We utilize Banzaicloud secret injection webhook to inject secrets in to running pods/containers. At the moment we currently have static AWS key pairs for each application and they can only be rotated manually. AWS provides the commercial solution by themselves which does automated AWS IAM User key rotation, but to save the costs we are trying to develop ourselves the solution that would do the exact same.
Flow would be like this:
Rinse and repeat basically. That is the idea. My main question is - Is it even possible to do it that way and would script be sufficient enough to perform and automate this ?
In short what I now by now Banzaicloud’s webhook only would read, wrap, transfer, decode and put the secret in to env variable when requested and f.e. if you try to retrieve it, it always comes back as a variable. So in the end I am not sure how this can be done correctly.
As well I want to point out that I’ve seen a feature request open in Github repository for vault for a possibility that Vault could rotate existing IAM user key pairs, but it has not been developed yet. So as we both know, currently engine creates the new IAM users with key pairs that have a defined TTL. As well we have already defined secret pathways in the applications that are used to inject secrets and basically a workaround would be copying newly created aws dynamic secret and writing it in existing vault pathway which overwrites old Key pairs and let’s application function without any disruptions in that way without us as well needing to have major changes in architecture.
So the main reason you are trying to do it this way is to reduce the amount of change needed to the existing applications? I’d strongly recommend looking at adjusting the apps to handle the AWS secret engine directly as the mechanism you detail feels quite complex & fragile. What happens if Vault expires a set of AWS credentials before the app timeout happens (as the app would have no link to the lease details)?
You also wouldn’t be able to handle lease renewals which will result is request failures - a set of credentials might expire in the middle of something the app is doing, meaning requests start failing. Even if the app timeout has expired as expected I’d imagine the existing app doesn’t handle the need to update credentials during processing.
If the app supports things correctly you’d generally look to renew leases rather than just letting them expire (up to a max TTL duration), with credentials created the first time they are needed/requested by the app and removed automatically once the app stops using them (e.g. if it is stopped). You’d have the lease duration being fairly short (say an hour) with the max TTL being longer (depending on your policy could be days or weeks). As the app would have full visibility of the lease it could ensure renewals happen early enough to make sure replacements don’t affect operations.
For the above you can replace “app” with your sidecar - many sidecar apps will fully manage all the AWS lease renewals & credential rotation. I don’t know if the one you are currently using does all this already.
Hi Stuart,
Could you please provide us with the example sidecar apps that can fulfil the AWS lease renewals & credential rotation. Currently, we are using the Banzai cloud secret injector as a sidecar to inject the secrets from the vault and this is not supported by the vault AWS secret engine backed to inject. It would be really great if you could point us in the right direction.