CA private key from Vault CA

Vault
1

Hi,
I’ve read through a few guides, I am trying to supply the Vault CA cert and private key to create a secret in Kubernetes as per this:

This shows how to generate said CA certificate:

However there is no mention of how to get the private key while generating the root ca cert nor the intermediate.
Commands such as this:

vault write -format=json pki/root/generate/internal \
 common_name="pki-ca-root" ttl=87600h | tee \
>(jq -r .data.certificate > ca.pem) \
>(jq -r .data.issuing_ca > issuing_ca.pem) \
>(jq -r .data.private_key > ca-key.pem)

Do not produce a key, only null.
The only time I can get private keys is when producing subdomain signed certs.
Does anyone know if it’s possible to get the root or intermediate key from a Vault CA?

2

Although you can use a Vault CA to generate a cert to use on Vault it isn’t as simple are you think it is.

Setting up the Vault CA is a lot more involved and needs to go through cross signing and Intermediate certs, etc… follow Build Your Own Certificate Authority (CA) | Vault - HashiCorp Learn

If you are going to use the Vault CA for other systems, then it’s worth it, if not then don’t waste time with it as it’ll be a beast to manage (any CA is). If you just need a cert for Vault then use openssl or letsencrypt to generate your cert depending on your situation.

Lastly, if you do want to use Vault CA, you have to setup and stand up your vault without an SSL, then setup CA/Intermediate, then generate your cert, then stop and reconfigure your Vault to use that cert.

3

I probably should have provided a bit more detail, I actually have an existing Vault in the cloud, which is the CA for our on-prem domain, I was looking to deploy a local Vault (not a CA) within our Kubernetes cluster using certificates from the cloud Vault CA (which provides many certs for our various applications with no issues).
It seems that Vault hides/consumes private keys on generation of root and intermediate certs, which hasn’t been an issue to this point, but since the setup of Vault in a Kubernetes environment requires a CA root private key along with the root cert, this is where the issue lies.
Perhaps the answer is to create a root cert and key from openssl and import this into the cloud vault somehow to use, instead of it generating it’s own.
Then proceed to use that cert and key for importing to any other vault installs that you need to stand up.

1 Like
4

Then AFAIK you’re using the wrong engine to try to create your server cert. I know enough to be dangerous here, so take this with some salt:

I believe, you generate server certs from your PKI engine not your PK engine. At least that’s what I use and this works great.

# vault write pki-int/issue/<server-cert-role> ttl=${ttl} common_name="${common}" ${altNames} ip_sans="${ipSans},127.0.0.1" -format=json > ${common}.certs.json"

Your ${common}.certs.json now contains your “certificate”, “private_key”, “issuing_ca”, “ca_chain” as well as your “serial_number” which you can use to track the cert in vault.

5

yes this is the way to issue certs for the various subdomains from the CA, sadly however it cannot issue the CA root certificate and key as I understand it.
This is what is required to be in the kubernetes secret as per the setup guide:

kubectl --namespace='vault' create secret tls vault-ca-crt --cert ./tls-ca.cert --key ./tls-ca.key

This provision of the CA root cert and key is what allows the new install of Vault to trust the certs produced further along for the cluster encrypted connections to support raft (or any other created certs by the original Vault CA)

6

I’m know even less about Kubernetes than I do about CA’s but from reading that command and what I have gleamed from other conversations, I think that’s actually importing your your CA cert into Kubernetes so that you can auto provision certs out of Kubernetes for your ingress(es).

Now if you have a cert (from Vault) and just need to assign it to an ingress, I believe this is what you need to do to import a cert into a kub secret so you can use it as on an ingress.

7

It’s never possible to get the private key out of PKI Certificate Authority. Nothing to do with Vault.

If you ever find yourself wanting to export the private key of a Certificate Authority, whatever make or model, you’re doing it wrong.

Your Vault CA could issue a subCA certificate to your Kubernetes cluster. You will end up with a two level PKI, a common practice but not every client is able to walk the chain, you might have to store Vault’s CA’s public key along with the Kubernetes subCA public key in your client’s various keystores.

8

I don’t think this is correct, it’s also bad policy to say “export” your key. I agree to be secure you don’t want to, or shouldn’t. but it’s certainly possible and can be done – the answer to that is you have to do when you generate your CA cert, after that it isn’t exportable anymore.

9

You can’t trust a key that’s going around. Once it is out, it’s out and it can’t be trusted. You could indeed export Vault’s private key, even import an existing key from some external system, but you get the trust that goes with a key that was seen outside the system: next to zero.

I’m told that Hashicorp is working on a HSM enabled PKI backend (beyond the current seal wrap) to enforce the “impossible to export” concept even further. The private key would never leave the HSM and the actual certificate signature would be done by the HSM, like many/most enterprise grade PKI already do.

10

That’s a fine argument of shouldn’t. But then you can’t say “Cannot”.

1 Like
11

Hi @ixe013 appreciate the feedback and yes it makes perfect sense that we shouldn’t be exposing keys that aren’t meant to be exposed, clearly bad security practice.

This public key sounds interesting, in the guide for Vault on Kubernetes, it does not actually specify public or private CA key, just “For PEM encoded TLS Certs where you have the key:”

So my next question then is, how do we obtain the public key from our Vault CA?
There is no mention in the tutorial on “Build your own CA” of exporting a key of any type, public or private.

12

google saves the day:

openssl x509 -pubkey -noout -in /root/ca/vault_root_ca.pem > pubkey.pem

I’ll go and test this and see if it works.

13

Looks like that is a no:
error: tls: failed to find PEM block with type ending in “PRIVATE KEY” in key input after skipping PEM blocks of the following types: [PUBLIC KEY]
Must be a private key.

14

There is a public endpoint for that in Vault, but you must use cURL for it (not vault read) because vault cli expects a JSON document.

curl $VAULT_ADDR/v1/pki/ca/pem --output vault-ca.pem

Of course your Linux configuration should already trust your CA :wink: . If not, add --insecure to the cURL command line.

If you need the DER (aka binary) version of the certficate, better to remove /pem at the end of the url instead of running the PEM through openssl.

curl $VAULT_ADDR/v1/pki/ca --output vault-ca.der
15

I find the comment so misleading, indeed, because it has everything to do with vault. If you generate the CA with openssl, you are going to have to have access to the private key in order to create that certificate.

I’m trying to find out if you can actually see the private key of the root CA in vault (for instance when you generate the root CA), but I still haven’t found a definite answer, although it seems that you cannot. Can anyone answer that?

So the point would be that you generate a root CA in vault, you cannot have access to the private key (for security reasons), but you can sign certificates with it whose private keys are going to be output only at the time of the issuing, right? Am I understanding this correctly?

16

Yes understand correctly. You will get best security by generating the CA key pair in Vault with either:

The private key is there, in plain text in memory and encrypted in Vault’s internal storage. Vault will use it to sign certificates, but there is no API to export the private key.

If you import an existing private key in Vault, it’s “too late” for security. It is impossible to guarantee the key was not compromised. This import feature should be seen as a way to move an existing insecure CA to Vault, and then use Vault’s APIs to do a key rotation in a secure way.

Bonus content:

You can take that security feature one step higher with the Enterprise version of Vault, where the key pair can be generated by a hardware security module (HSM), so that even Vault itself never sees the private key (it will forward signature operations of certificates, OCSP and CRL to the HSM).

1 Like
17

This is true… but just to give everyone clarity that it’s really not that hard to grab the key if you have admin access to Vault:

Option 1

If you can turn raw_storage_endpoint on in the Vault configuration file, and have access to define Vault policy, it’s quite easy to fetch the private key out of the sys/raw/ API.

Option 2

If you have access to manage plugins on a Vault server, you can make a forked version of the PKI secrets engine that adds a new API for getting the CA private key, and load it into a running Vault server.

1 Like
18

Thanks @maxb, TIL how to extract the private key from the PKI backend. For the record, here is how it’s done. Requires that you enabled the raw api (dangerous) or that you run a dev server:

  1. Find the UUID of the PKI backend with PKI_MOUNT_UUID=$(vault read --format json sys/mounts | jq -r '.data[] | select(.type=="pki").uuid')
  2. Find the private key UUID with PKI_PRIVATE_KEY_UUID=$(vault list --format json sys/raw/logical/${PKI_MOUNT_UUID}/config/key | jq -r .[0])
  3. Extract the private key with vault read --field value sys/raw/logical/${PKI_MOUNT_UUID}/config/key/${PKI_PRIVATE_KEY_UUID} | jq -r .private_key

Adjust the first command (or make it a loop) if you have multiple PKI mount, like if you have Vault both the root CA and issuer.

$ PKI_MOUNT_UUID=$(vault read --format json sys/mounts | jq -r '.data[] | select(.type=="pki").uuid')
$ PKI_PRIVATE_KEY_UUID=$(vault list --format json sys/raw/logical/${PKI_MOUNT_UUID}/config/key | jq -r .[0])
$ vault read --field value sys/raw/logical/${PKI_MOUNT_UUID}/config/key/${PKI_PRIVATE_KEY_UUID} | jq -r .private_key
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAuYXQupxfiL/BZBEs7Mhp/F/b9qMhaorohgKcN2W2XQ0K7tRB
r/o9Ft9rFbrpEe26twSOSgoEal8MLNwcEUQM6Qfw7yVEa7yUWHyxM3XprZfhkNMR
b5I0e+n9Q4IUjooxp0zxF7kYpU3YZpDYC92YEmaxj3uGKag0mwllruKwwvct8WFi
NBur7K+6+jThm69WU87tUu+2iI4Dnokt4NKtEir1HZlO+q35lxZ1HFqIFttHLojQ
ymwIzFjYxmrgJTPO9B1V9J2cOP1N93bxtyy8k/cOnSAHxoy37QfVr71Tp2JkQ72s
R2I4tYAp1qepAjw+rFCVEYrCQHkBm6eX9l9iVwIDAQABAoIBAEyWaH/RNEMw3jZg
PMqXReGw9Q6Rc6cXJzzDcRgRfkJVsxC7QU7ToEqD+yIiYjtLh8wsLk10azQ6fr06
NMAP9sBwzc779I4BcMBC4ahb4hcO30i5SbZkhhQtmac32SKq/CqTwefIOttrEo3s
97OWTEQ0btLL7Kvfy6/BJmAXJ1+klSTYsDLWuj+jpWLCb7lY2+H8oRumgfLlxtG9
P0lJ0UKwBnAd6eOnRM3mS+/CuhVGyyv8kF6P1+nyEhX7lL4zftYfR04lDAmlnx0z
bWBwixGRNvtWEF+/HRYcXER0WZFaZz+/qeXqfLpnmPGVyLorDla0rWL/kdILuBBF
nbBgxeECgYEA0xHetKkmcG+oHwRssGDZxw55p3OzDWWaylKzCHQAm7YbJlgZBhRz
sXJ9C71zF6W4B6bEbpEbU0MF73qYMmbvXgtRGJni464wB6E9A8NHrXshkjseDnX0
pIxrOVWVo4X3d8n07DD4BKQaqU+UbCPm47RT1oouBEUTZUYc9kHo24cCgYEA4QPF
bu+Uagxm7maEM6d5SDOf3NRUprNuXh5GfaBvmVE5jcPI/gdrairfKyWOixP5Vfeh
v6cL5TzGVylO2Cent9nkzKyox19ThHBf7Jw/QeObOMxtwEwfjqFFsmHGli2zK+PW
YRdKi9S4xCSREpxYli0NGoElYWXhJfFvNebuFrECgYAOAaiSuorXeUFh1YxU5Ahv
ojFGgHIIWI6+EGDbFb+YlawVjlESyzamNt+/cLH7hr7Amzt7zttOWitBqhQBfD00
M3INPOEGlN+REg0Xe9T82SfwKUYkVYWHid3vrxNAB7rVUyySo+lBfZZbV/GUf4lp
4OSTXxYQqHjSBqYTXWpTbwKBgBRCr6tOfTGTTk1XA75W7Q+4PkO3Bqw/6ccluLkl
EgdfDeya3WUqK9zSYWfUKOKS+NdpxbGpC+QnNcCxYn0KDQr73qMjRMs25SE+sRRv
+S4onVhe597xu8Gu+cSXEzeAg+qGyE3TuA2hKMXYeQuQ8lULV2u0hzYe40f0Vkwn
L/bxAoGATC3AFijGidGQ/Y9Chq1zm4XCKsr8ZiIj4hnOfQ09yRvZI6xW9WDG4lN9
9Dr9ZRbhRNHeAX+41B8Ub6ChnIOkArhka6nuFpQd7SuWc86OqNpvo46w7gYIotMl
5gtt9AomFde8FmPztpCjKLmCg+hIbhxC5lK4W8soLsai/9aCdHk=
-----END RSA PRIVATE KEY-----
2 Likes
19

Is type=exported, i.e., /root/generated/exported not sufficient here? This has been mentioned in the API docs.

If you’re explicitly generating the CA to use externally, it seems to be what the OP wanted. For migration after the fact, I do generally agree that its definitely more ideal to consider how to move keys properly, but /sys/raw works in a pinch.

20

i am getting empty response to the below command {} please help?

vault list --format json sys/raw/logical/${PKI_MOUNT_UUID}/config/key | jq -r .[0]