How is the PKI CA chain generated?

While following this tutorial, I was surprised to see that the new root (root-2024) issuer’s ca_chain field changes when the cross-signed intermediate issuer is created, even though there were no write operations to this issuer.

At the same time, the cross-signed intermediate issuer only includes the cross-signed intermediate and the old root, and not the new root that uses the same key as the cross-signed intermediate.

I have two questions about this behaviour:

  1. How does vault decide which certificates are included in the ca_chain field?
  2. Is this different for root certificates and intermediate certificates?

Left to its own devices, I think Vault tries to figure out a useful chain of connections, amongst all the issuers present within the PKI secrets engine.

I assume that’s what you’re seeing - that once something that connects to the new root exists in the same PKI secrets engine, it shows up in the ca_chain.

There is a manual_chain option for when Vault’s best guess is not what you wanted it to serve: https://developer.hashicorp.com/vault/api-docs/secret/pki#manual_chain

Yes - I was hoping someone could shed some light on what the algorithm is when manual_chain is not used. How does Vault decide on what certificates to include in ca_chain, and where does it look for them?

Well, I basically already answered that, if not in very much detail.

By “useful chain of connections”, I mean looking at subject and issuer distinguished names, and signatures, i.e. the normal way chaining of X.509 CA certificates is analysed.

I am not just using “issuer” in a generic sense here, I am referring to the specific concept of an “issuer” as a piece of configuration of the PKI secrets engine, which is mentioned extensively in its API docs.

If you want detail on a level beyond this, you’d have to turn to the source code (as would I): https://github.com/hashicorp/vault/blob/main/builtin/logical/pki/chain_util.go

(sorry to necro old topics; I do not check the discuss forum that often)

One commentary is that chain building only applies from an Issuer on up: we do not re-build chains for every possible (equivalent) issuer of a leaf certificate on issuance.

This means that cross-signed intermediates directly used to issue intermediates are not detected via automatic chain building, since they are peer issuers. A manual chain must be constructed to pull in these peer certificates and their parents, if any.

If automatic chain building with cross-signed intermediates is desired, I’d recommend adding a signing CA under the cross-signed ICAs, (increasing the chain depth by one).

Additionally, this is strictly issuers within the mount, we do not check roots/… from other mounts.

1 Like

I am more used to read you on GitHub, that’s true :wink: @cipherboy.

I am not sure to understand what you mean so if you get on this topic once again I hope you can clarify for my dumb brain.

I have been reading about /intermediate/cross-sign for hours now but I can’t get my head around it. The tutorial step 8, the rotating primitives, the API and GitHub issues too.

From what I understand, cross-signing use the same key material to generate two certificates signed by highers CAs in the hierarchy. In doing so there now several chains of trust but they are not really forking but are rather crossing because one CA is present with the same key material in two different chains. Please correct me if I am wrong.

When it comes to Vault this is where I am lost.

Let’s say I have a three layer PKI with the following mounts from root to lower : pki_root, pki_int and pki_iss.

In pki_root, I have newRoot and oldRoot issuers as root CAs.

The documentation and other references detail how to cross-sign the new root with the old root in pki_root/intermediate. This new cert uses the same key material as newRoot.

So once this is done, should I sign a new issuer in pki_int with the cross-signed certificate or new Root in order to have all clients with the oldRoot or the newRoot installed to trust the certs lower in the pki ?

As the newRoot chain is augmented now with the chain to the old root, I think with no certainty that it should work just like this.

But what about manual chain as the doc says to edit in a way that oppose what the rotating primitives and tutorial step 8 tell ?

I swear once I understand how to do this properly, I will do a PR on this because I think we are numerous wondering how we can do the Root rotation.

Thanks for raising this. I’m experiencing exactly the same confusion as you are.

Like you, when following the tutorial I was surprised to see the new root (root-2024) is changed. But the ca chain for the cross signed intermediate only appears to contain itself and the old root. If I request for a new certificate to be generated using the intermediate with: vault write pki_int/issue/example-dot-com common_name="test.example.com" ttl="24h", I don’t see the new root in the returned CA chain. I had assumed for the rotation to have worked correctly, the new root would need to be present.

I’m wondering what I should be looking at in the returned payload to understand how the certificate requests now link back to the new Root CA.

I see that I am not alone in my struggling :smile:.

According to me and you should probably not trust it too much. It should concern only internal Vault pki not imported ones but i may be wrong.

Within pki_root :

  • root/oldRoot
  • intermediate/cross-signedRoot (newRoot by oldRoot) with ca_chain = oldRoot -> cross-signedRoot. I think this one is built automatically.
  • root/newRoot with ca_chain = oldRoot -> cross-signed -> newRoot. This chain is built automatically as the tutorial step 8 states.

Now you should sign a new Intermediate CA in pki_int :

  • intermediate/oldICA with ca_chain = oldRoot -> oldICA
  • intermediate/newICA should be signed by newRoot. There i should test it but according to me we must have oldRoot -> cross-signed -> newRoot -> newICA. Maybe we have to set it manually because we want the oldLeaf to trust the new Root.

I will build it and get back here to be sure. And we have to make sure that cross-signed and newRoot have the same Subject

Ah, I see, thanks for flagging this @Malshtur and @luke.stephenson – this seems to be an error in the tutorial headings.

The tutorial step 8 describes creating a root bridge certificate – an “intermediate” (in technicality only) that is useful for bridging between two separate roots of trust. E.g., if you had devices with root-2023 hard-coded and not replaceable, this bridge CA would let you validate against it, in a slightly different way than cross-signing would.

Sometimes this is useful for injecting a new root CA into a hierarchy (temporarily) to separate PKIs. E.g., if everything chained up through a common widely-trusted root CA but now we wish to separate CAs, a temporary bridge CA can be used here to allow people still trusting the old root (but not yet moving to the separate PKI) to access this.

I’ll follow up with the tutorials team, thank you!


I think perhaps an alternative (correct under this header name) would be:

  1. Get a CSR to cross-sign the intermediate
$ vault write -format=json pki_int/intermediate/cross-sign \
      common_name="example.com Intermediate Authority" \
      key_ref="$(vault read pki_int/issuer/example-dot-com-intermediate \
      | grep -i key_id | awk '{print $2}')" \
      | jq -r '.data.csr' \
      | tee cross-signed-intermediate.csr
  1. Sign the CSR under the new root.
$ vault write -format=json pki/issuer/root-2024/sign-intermediate \
      common_name="example.com Intermediate Authority" \
      csr=@cross-signed-intermediate.csr \
      | jq -r '.data.certificate' | tee cross-signed-intermediate.crt
  1. Import the cross-signed certificate into the new mount.
$ vault write pki_int/intermediate/set-signed \
      certificate=@cross-signed-intermediate.crt
$ vault write pki_int/issuer/<uuid> issuer_name=xc-example-dot-com-intermediate
  1. When reading issuers, the CA chain will not change:
$ vault read pki_int/issuer/example-dot-com-intermediate
$ vault read pki_int/issuer/xc-example-dot-com-intermediate
  1. But if it is desired to update them to refer to each other on sign requests, use manual_chain:
$ vault patch pki_int/issuer/example-dot-com-intermediate manual_chain=self,xc-example-dot-com-intermediate
$ vault patch pki_int/issuer/xc-example-dot-com-intermediate manual_chain=self,example-dot-com-intermediate

and then they would update their chains and do what we’d expect. This is since they are sibling CAs, not in a hierarchy, so automated chain building does not detect them.

Thanks for the details @cipherboy !

If I may give my opinion, I would keep both in the tutorial but clearly explain what are their respective purposes.

The root bridge is, as you said, a nice way to rotate root by defining a clear timeframe (pki/intermediate ttl) to handle root rotation in devices. This inserts the new root in the existing chain for transition. A bit like a join to split later.

The cross-signing procedure you described would rather be for joining two trust chains together at the pki_int/intermediate level. It seems like the opposite of the root bridge.

I do not think both should be in the tutorial, because they are not both practically useful.

The version already in the tutorial covers cross-signing of a new root CA by its predecessor, which is a common useful technique for rotation.

The version @cipherboy produced to illustrate a difference, is not part of what a CA administrator should actually do in any circumstance I can think of - cross-signing an old intermediate with a new root is not a normal thing - you would usually take the opportunity to generate new previously unused key material.

It would be better to fill in the missing steps in the tutorial needed to complete a normal migration process - once you have the new root, and its cross-signed form, you then need to make a new intermediate CA, and import the cross-signed new root into the pki_int secrets engine so it is available in the chain served along with leaf certificate issuance.

\o hey @maxb :slight_smile:

I think if the intermediate CA were not an old intermediate, but instead a_ new_ one, this would be more common than (and perhaps preferable to) cross-signing root CAs. In either case, you still need to distribute the bridge and/or the cross-signed ICA pair.

But duly noted about finishing the rotation steps as well.

  • A

You are right @maxb , completing the tutorial with the necessary steps will clarify all of this for a lot of people.

For correctness, i would also update the imported issuer with a name to ease maintenance (optional but i like to keep name on it to remember this is the root or intermediate import) and also update the usage with read-only. But as they don’t have a private key in the mount where they are imported, it may not be necessary.

@cipherboy can I open a PR for the tutorial or a part of it ?

@Malshtur Sadly the tutorials repository is not public, but otherwise I’d happily welcome a PR. :confused:

@cipherboy Oh :confused: well maybe it could fit somewhere else otherwise I will try to share it somehow.

Is there any reason it needs to remain that way?

So what is messed up in that tutorial, which part? As I don’t want to spend time on something that’s not up to date.