TLS Auth: invalid certificate or no client certificate supplied with Puppet CA

[ahmet2mir]

Hello friends,

I’m trying to debug TLS Auth using Puppet CA SSL certificates.

We have a Root / Intermediate / Leaf PKI for Puppet CA.

Each node got signed by the Intermediate, and everything goes well on Puppet side.

We need to enable Vault TLS auth using Puppet CA but I have the error “invalid certificate or no client certificate supplied”

I’m trying to reproduce the issue using a different PKI architecture.

Start the server (on Linux, need Vault CLI & Step CLI)

#!/bin/bash

command -v step 2>&1 >/dev/null || {
  echo "Missing command 'step', download it and ensure it's in your PATH"
  echo "https://dl.step.sm/gh-release/cli/gh-release-header/v0.21.0/step_linux_0.21.0_amd64.tar.gz"
}

command -v vault 2>&1 >/dev/null || {
  echo "Missing command 'vault', download it and ensure it's in your PATH"
  echo "https://releases.hashicorp.com/vault/1.9.2/"
}

mkdir -p "${PWD}/data/vault"
mkdir -p "${PWD}/data/ssl/"{vault,client}

[ ! -f "${PWD}/data/ssl/vault/VaultCA.crt" ] && {
  echo ">> Generate Vault CA and a signed cert"

  echo password > "${PWD}/data/ssl/vault/password-file"

  step certificate create \
    --profile root-ca \
    --password-file "${PWD}/data/ssl/vault/password-file" \
    "VaultCA" \
    ${PWD}/data/ssl/vault/VaultCA.crt \
    ${PWD}/data/ssl/vault/VaultCA.key

  step certificate create \
    --profile leaf \
    --ca ${PWD}/data/ssl/vault/VaultCA.crt \
    --ca-key ${PWD}/data/ssl/vault/VaultCA.key \
    --no-password --insecure \
    --ca-password-file "${PWD}/data/ssl/vault/password-file" \
    --san "localhost" \
    --san "127.0.0.1" \
    "localhost" \
    "${PWD}/data/ssl/vault/vault.local.crt" \
    "${PWD}/data/ssl/vault/vault.local.key"
}

echo ">> Generate Vault config"

tee "${PWD}/data/config.hcl"<<EOF
storage "raft" {
  path = "${PWD}/data/vault"
  node_id = "local"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "${PWD}/data/ssl/vault/vault.local.crt"
  tls_key_file = "${PWD}/data/ssl/vault/vault.local.key"
}

api_addr = "https://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true
disable_mlock = true
EOF

echo ">> Starting vault in front"
vault server -config "${PWD}/data/config.hcl"

The server start, then I could bootstrap it.
I generate the same pki than our Puppet and add the intermediate on vault side

#!/bin/bash

# https://www.vaultproject.io/docs/auth/approle

command -v step 2>&1 >/dev/null || {
  echo "Missing command 'step', download it and ensure it's in your PATH"
  echo "https://dl.step.sm/gh-release/cli/gh-release-header/v0.21.0/step_linux_0.21.0_amd64.tar.gz"
}

command -v vault 2>&1 >/dev/null || {
  echo "Missing command 'vault', download it and ensure it's in your PATH"
  echo "https://releases.hashicorp.com/vault/1.9.2/"
}

VAULT_ADDR='https://127.0.0.1:8200' 
VAULT_CACERT="${PWD}/data/ssl/vault/VaultCA.crt"

export VAULT_CACERT VAULT_ADDR

[ ! -f "${PWD}/data/init.json" ] && {
    echo ">> Init vault"
    vault operator init -format=json > "${PWD}/data/init.json"
    echo
}

echo ">> Init content"
jq . "${PWD}/data/init.json"
echo

echo ">> Unseal vault"
vault operator unseal $(jq -Mcr '.unseal_keys_b64[0]' "${PWD}/data/init.json")
vault operator unseal $(jq -Mcr '.unseal_keys_b64[1]' "${PWD}/data/init.json")
vault operator unseal $(jq -Mcr '.unseal_keys_b64[2]' "${PWD}/data/init.json")
echo
until vault status | grep -q 'Active Since'
do
    sleep 1
    echo ">> Waiting status up"
done

sleep 1

echo ">> Login root"
vault login $(jq -Mcr '.root_token' "${PWD}/data/init.json")
echo

[ ! -f "${PWD}/data/ssl/client/Root.crt" ] && {
    echo ">> Generate Client CA and a signed cert"

    mkdir -p ${PWD}/data/ssl/client

    echo password > "${PWD}/data/ssl/client/password-file"

    step certificate create \
      --profile root-ca \
      --password-file "${PWD}/data/ssl/client/password-file" \
      "Root" \
      ${PWD}/data/ssl/client/Root.crt \
      ${PWD}/data/ssl/client/Root.key

    step certificate create \
      --profile intermediate-ca \
      --ca "${PWD}/data/ssl/client/Root.crt" \
      --ca-key "${PWD}/data/ssl/client/Root.key" \
      --password-file "${PWD}/data/ssl/client/password-file" \
      --ca-password-file "${PWD}/data/ssl/client/password-file" \
      "Int" \
      ${PWD}/data/ssl/client/Int.crt \
      ${PWD}/data/ssl/client/Int.key

    # enforce key usage to the same than Puppet CA
    tee ${PWD}/data/ssl/client/leaf.json <<EOF
{
    "subject": {
        "commonName": "localhost"
    },
    "keyUsage": ["digitalSignature", "keyEncipherment"]
}
EOF
    step certificate create \
      --template "${PWD}/data/ssl/client/leaf.json" \
      --ca ${PWD}/data/ssl/client/Int.crt \
      --ca-key ${PWD}/data/ssl/client/Int.key \
      --no-password --insecure \
      --ca-password-file "${PWD}/data/ssl/client/password-file" \
      "localhost" \
      "${PWD}/data/ssl/client/client.local.crt" \
      "${PWD}/data/ssl/client/client.local.key"
}

echo ">> Creating policy 'policy'"
tee policy.hcl <<EOF
path "*" {
  capabilities = [ read", "list" ]
}
EOF
vault policy write policy policy.hcl

# > Success! Uploaded policy: policy

echo ">> Enable TLS auth with Root"
vault auth enable -path="mytest" cert
vault write auth/mytest/certs/intca \
    display_name=mytest \
    policies=policy \
    certificate=@${PWD}/data/ssl/client/Int.crt \
    ttl=3600
vault read auth/mytest/certs/intca

Once bootstrap done, I run a client test

command -v vault 2>&1 >/dev/null || {
  echo "Missing command 'vault', download it and ensure it's in your PATH"
  echo "https://releases.hashicorp.com/vault/1.9.2/"
}

VAULT_ADDR='https://127.0.0.1:8200' 
VAULT_CACERT="${PWD}/data/ssl/vault/VaultCA.crt"

export VAULT_CACERT VAULT_ADDR

echo ">> Test TLS with ClientCA signed cert"
vault login \
    -method=cert \
    -path=mytest \
    -client-cert="${PWD}/data/ssl/client/client.local.crt" \
    -client-key="${PWD}/data/ssl/client/client.local.key" \
    name=intca

Vault login without issue.

Now I change the Int.crt by our Puppet Int, and also client.local.crt and client.local.key by puppet signed certs.

When I run the bootstrap again to override the auth certificate then the client, but now I have invalid certificate or no client certificate supplied

Then I remember some issues we had with jruby/openssl (Puppet CA is written in ruby and run with jRuby) and especially AuthorityKeyId and SubjectKeyId

• Certificates generated using jruby-openssl have a bad value for Authority Key Identifier, CRL Distribution Points and Authority Information Access · Issue #994 · jruby/jruby · GitHub
• create_extension('subjectKeyIdentifier', 'hash') uses the wrong key id · Issue #173 · jruby/jruby-openssl · GitHub

So to I suspected this condition to fail, so I used the same approach than vault to verify

package main

import (
	"crypto/x509"
	"encoding/pem"
	"fmt"
	"encoding/base64"
)

func main() {
	var leafPem = []byte(`
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
`)

	var intPem = []byte(`
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
`)
	var rootPem = []byte(`
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----`)

	rootBlock, _ := pem.Decode([]byte(rootPem))
	intBlock, _ := pem.Decode([]byte(intPem))
	leafBlock, _ := pem.Decode([]byte(leafPem))

	rootCert, err := x509.ParseCertificate(rootBlock.Bytes)
	if err != nil {
		panic("failed to parse certificate PEM")
	}

	intCert, err := x509.ParseCertificate(intBlock.Bytes)
	if err != nil {
		panic("failed to parse certificate PEM")
	}

	leafCert, err := x509.ParseCertificate(leafBlock.Bytes)
	if err != nil {
		panic("failed to parse certificate PEM")
	}

	fmt.Printf("Name %s\n", rootCert.Subject.CommonName)
	fmt.Printf("Name %s\n", intCert.Subject.CommonName)
	fmt.Printf("Name %s\n", leafCert.Subject.CommonName)

	// https://github.com/hashicorp/vault/blob/v1.9.2/builtin/credential/cert/path_login.go#L89
	rootSKI := base64.StdEncoding.EncodeToString(rootCert.SubjectKeyId)
	intAKI := base64.StdEncoding.EncodeToString(intCert.AuthorityKeyId)
	intSKI := base64.StdEncoding.EncodeToString(intCert.SubjectKeyId)
	leafAKI := base64.StdEncoding.EncodeToString(leafCert.AuthorityKeyId)

	if rootSKI == intAKI {
		fmt.Printf("root/int match\n")
	} else {
		fmt.Printf("root/int don't match\n")
	}

	if intSKI == leafAKI {
		fmt.Printf("int/leaf match\n")
	} else {
		fmt.Printf("int/leaf don't match\n")
	}
}

When I run it

$ go run main.go
Name Puppet Root CA
Name Puppet CA Intermediate
Name mynode.somewhere.lan
root/int match
int/leaf match

To ensure that it’s not a “chain” issue (even if it works locally with a local root/int/leaf pki),
I run tests as (and all returns the same error message)

vault role	client leaf	client leaf+int	client leaf+int+root
root
int
root+int

I don’t have more details (no logging on path_login) when enabling server with -log-level=trace or vault debug

The Puppet leaf (also note that we use custom OID in extensions in order to allow a better trusted identification)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: xxxx (0xxxxx)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA Intermediate
        Validity
            Not Before: Sep 29 01:34:17 2021 GMT
            Not After : Sep 29 01:34:17 2026 GMT
        Subject: CN=mynode.somewhere.lan
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:xxxxx:6d
                Exponent: yyyy (0xyyyy)
        X509v3 extensions:
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                keyid:xx:38

            X509v3 Subject Key Identifier: 
                xxx:E1
            1.3.6.1.4.1.34380.1.2.1.1: 
                ..test
            1.3.6.1.4.1.34380.1.2.1.2: 
                ..emea
            1.3.6.1.4.1.34380.1.2.1.3: 
                ..eu
            1.3.6.1.4.1.34380.1.2.1.4: 
                ..par1
            1.3.6.1.4.1.34380.1.2.1.5: 
                ..e1
            1.3.6.1.4.1.34380.1.2.1.6: 
                ..sandbox
            1.3.6.1.4.1.34380.1.2.1.7: 
                ..sb
            1.3.6.1.4.1.34380.1.2.1.8: 
                ..sandbox
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
         c1:xxx:90

So I’m a bit lost, my certs are parsed without issue, Subject/Authority key match, go could read them without other issues
but I have this error.

Any advice to help debug ?

PS: using currently vault v1.9.2 but same issue with v1.11.1

[barskern]

Just wanted to mention that I’ve been stuck on the same error as you. Though weirdly I managed to login with the certificate generated on the puppet master node (though not from any other node).