What are some ideas or options to consider in this scenario? Normally I’m used to keeping encrypted values in the repo, so life is a bit different now.
I want to determine how best to setup a vault structure between deployments with a dev, prod_green, and prod_blue type scenario. It seems like the simple build test deploy paradigm breaks down a bit if the values used to produce AMI’s are not in the repository though ( and in vault instead ).
Thats because If we build AMI’s in dev, with vault vars stored in a /dev mount, test them, and release the ami’s into production, what concerns me is that the value’s used to produced the images could be destroyed, making the AMI potentially difficult to reproduce. The values also aren’t tied to the repo commit.
I’m going to share some ideas for possible solutions. One I don’t like:
Alternatively we could
- Build ami’s in dev, test them. If they pass…
- Transfer the vars into a /prod_green or /prod_blue mount, build again, and deploy.
This would mean the values used to produce a current green or blue production deployment would be easier to use to reproduce a result, but this is a bit shabby because we are building the ami’s twice to ensure consistency, when we ideally should only do it once where possible. It delays the ability to iterate and release to production.
One idea I do like, with a question:
- Can vault do snapshots? perhaps we could use a snapshot and replicate all values into a some kind of unique root name? We could commit a vault snapshot of a tree (random pet / hash / id) into a repository. Each snapshot would contain everything needed for both dev and prod green/blue. We might have a structure in the mount as follows:
snapshot-random-pet/dev or base (most vars, even for prod are actually contained here)
snapshot-random-pet/green (some prod vars only, like overrides)
snapshot-random-pet/blue (some prod vars only, like overrides)
I like this approach because it produces version controlled results that can be immutable.
- when building, all variables would be pulled from the snapshot. When deploying to green or blue, only some of the values in that snapshot would pickup from some other path contained in that snapshot.
- if dealing with a bug in production, the new dev environment to test would be replicated based on the snapshot used for that commit.
- some exceptions to immutability might occur, like passwords that may be rotated in the kv store.
I like this idea, hopefully its not too complicated to do in reality. Snapshots could be automatically trashed after some time unless deployed, keeping junk to a minimum, but still, the values aren’t in the repo, over time, a commit hash may not reproduce a deployment. Perhaps it sounds silly, but would there be any value to AES encrypting the value store and keeping it in the repo (as a storage backend?), then restoring it back to the vault to use? It’s a bit like the old ansible encryption workflow I’m used to, but at least vault can manage the keys and would be the only thing decrypting the data.
Another idea, not so good:
- Any vault values that need to be used for an ami are copied into a seperate mount (eg /dev/random_pet.
- An AMI packer build can only read from that mount, no where else.
- Once the ami’s are produced, thay tagged with the random pet / or some id.
This is just ok, doesn’t provide any guarantees, not a fan of this really.
I’d like to know what others do out there though, its a bit of a hairy problem that it would help me to establish a good workflow early with.