The problem I’m facing is I don’t understand how this is supposed to work. The Vault PKI issue API docs at https://developer.hashicorp.com/vault/api-docs/secret/pki#generate-certificate-and-key have a warning that states The private key is not stored. If you do not save the private key from the response, you will need to request a new certificate.. This aligns with the observed behavior of the endpoint where each POST request generates a new cert.
So, the first thing I don’t understand, and this may be based on a faulty core assumption, is how does that work with multiple with secret blocks? My understanding is each with secret block is independent, so each with secret block will do it’s own HTTP post, which will generate a new certificate. As such the key and cert will not match. Is that not true? Does Consul-template cache/deduplicate the with secret blocks? What does it use as the common key? How does that work with the two different sets of args in the example (CA block has fewer args)?
Secondly, how does this handle refreshing the cert? Will this simply generate new certs each time the template is evaluated? My understanding with secrets like those from the kv engine is Consul reads the secret out of Vault regularly and updates the template when the secret changes. Is this another false assumption? If not how does the template know when it’s time to update the cert?
It’s a fun trick that’s… not very well documented I believe, but a {{ with secret }} call that has identical parameters (and an identical path) is rolled into a single request, where the response is then used to populate however many template blocks use that particular secret.
As far as refreshing goes: yes, any time the template is re-rendered (i.e. consul-template is restarted) it will just issue a new certificate. If you have consul-template running as a daemon it will track the lease TTL and will refresh the secret at something like 90% of it’s TTL, and will then write out a newly rendered template (and processes any on change events you have set up).
For certificates there is now another option I believe, {{ with pkiCert }} (see consul-template/docs/templating-language.md at main · hashicorp/consul-template · GitHub) that makes a single call and returns cert key and ca in one shot, but it will also check the existing certificate for validity and will only fetch a new one if it’s close to expiring so the end result is that you get fewer calls to Vault - mainly because you aren’t re-fetching a certificate any time consul-template runs.
So, for the pattern above, switch to using the pkiCert function instead, it makes life easier
a {{ with secret }} call that has identical parameters (and an identical path) is rolled into a single request, where the response is then used to populate however many template blocks use that particular secret.
If you have consul-template running as a daemon it will track the lease TTL and will refresh the secret at something like 90% of it’s TTL, and will then write out a newly rendered template (and processes any on change events you have set up).
Is this a special case handling of certs, then? The with secret handler appears pretty generic, does it detect that it’s referencing a pki mount type? How does it know it’s fetching a cert?
For certificates there is now another option I believe, {{ with pkiCert }}
Yea that looks a lot better, thanks for pointing that out. I like that the pkiCert endpoint documents a lot of my “How does this work?” questions. I’ll move in that direction.
I’m not sure to be honest, as far as I know the arguments have to be identical but it may be it just looks at the CN and TTL fields; but even if it would generate 2 calls the first 2 have identical parameters so your cert/key match, the 2nd one discards whatever cert is generated and just uses the CA field (which shouldn’t have changed between calls
It doesn’t, that’s why pkiCert was added a while ago; in some cases it doesn’t matter much, we still generate a ton of certificates using {{ with secret }} because the certificates are not stored into Vault, and no leases are generated - and they’re very short lived (4 hours) so it really doesn’t impact performance in any way.
Also if you use things like Nomad, a job that runs there is pretty much considered a transient thing, so there is no real difference between secret and pkiCert because they will both generate a new certificate - most of the time.
Excellent!
Scroll down a little further in the docs, there is a filter (I believe off the top of my head writeToFile) that can be used like {{ .Cert | writeToFile "foo.cert" }} where you can use the single template to write 3 different files (if you want) - so the actual template that is executed generates no output. The nice thing is though that any on change handling is done based on the executed template, so you get some form of atomic update where cert/key/ca is updated, and then the on change is executed.
The writeToFile function can be used with pkiCert to write your Key and Certs to separate files from a template.
{{- with pkiCert "pki/issue/my-domain-dot-com" "common_name=foo.example.com" -}}
{{ .Cert }}{{ .CA }}{{ .Key }}
{{ .Key | writeToFile "/my/path/to/cert.key" "root" "root" "0400" }}
{{ .CA | writeToFile "/my/path/to/cert.pem" "root" "root" "0644" }}
{{ .Cert | writeToFile "/my/path/to/cert.pem" "root" "root" "0644" "append" }}
{{- end -}}
Okay, neat. I think I need to go take this for a spin around the block and see how she handles, this looks like a better pattern than the one we’re currently using. Thanks for the guidance, it’s tipped the scale for me from “this looks good on paper but I have major questions” to “this looks like a superior pattern worth trying out.”
Never not experiment - that goes for just about everything. In my experience, theory is nice, but the “real world” is where it’s at. We always spend more time messing around with things than we do pondering the theory. I mean, we do look at the theoretical, but sometimes it’s just more fun to do some practical experiments and find out whether the thing that looked so good in theory actually works. Or vice versa where the theory on something is “meh” but then practically it works out so well it sort of makes the decision for you
Edited to add: for instance in Nomad, we still use secret as opposed to pkiCert for certificates, because any time a job is started (or restarted), it’s just easier to generate a new certificate, and since they’re short-lived and non-leased, it’s all good. So theoretically pkiCert looks better but practically it turned out it isn’t. On the flip side of that coin we do have a few cases where we’ve switched because it results in statistically significantly fewer calls to Vault.
