Following pki-acme-caddy example, but I'm getting "tlsv1 alert internal error" messages from curl

Vault
1

Folks,

I’m following the example at Manage certificates with ACME clients and the PKI secrets engine | Vault | HashiCorp Developer but I’m having some problems with caddy and curl. I get down to the steps at Manage certificates with ACME clients and the PKI secrets engine | Vault | HashiCorp Developer and curl comes back with the error message:

curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error

Here’s the debug output:

curl -v    --cacert "$HC_LEARN_LAB"/pki/root_2023_ca.crt     --resolve caddy-server:443:127.0.0.1     https://caddy-server
* Added caddy-server:443:127.0.0.1 to DNS cache
* Hostname caddy-server was found in DNS cache
*   Trying 127.0.0.1:443...
* Connected to caddy-server (127.0.0.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /home/apldadmin/learn-vault-pki/pki/root_2023_ca.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error

Now, looking into this a bit more, it seems that the problem is that Caddy is starting up with an ACME server reference, but without a local copy of the Root CA Cert. I see messages like this in the output of docker logs:

{"level":"error","ts":1748903530.5748394,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"caddy-server","issuer":"172.12.12.100:8200-v1-pki_int-acme-directory","error":"http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)"}
{"level":"error","ts":1748903530.5748577,"logger":"tls.obtain","msg":"will retry","error":"[caddy-server] Obtain: http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)","attempt":1,"retrying_in":60,"elapsed":0.000281059,"max_duration":2592000}
{"level":"info","ts":1748903530.5767524,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1748903530.5767615,"msg":"serving initial configuration"}
{"level":"info","ts":1748903590.5750046,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"caddy-server"}
{"level":"error","ts":1748903590.575275,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"caddy-server","issuer":"172.12.12.100:8200-v1-pki_int-acme-directory","error":"http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)"}
{"level":"error","ts":1748903590.5753124,"logger":"tls.obtain","msg":"will retry","error":"[caddy-server] Obtain: http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)","attempt":2,"retrying_in":120,"elapsed":60.000735636,"max_duration":2592000}
{"level":"info","ts":1748903710.5760508,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"caddy-server"}
{"level":"error","ts":1748903710.5763116,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"caddy-server","issuer":"172.12.12.100:8200-v1-pki_int-acme-directory","error":"http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)"}
{"level":"error","ts":1748903710.576335,"logger":"tls.obtain","msg":"will retry","error":"[caddy-server] Obtain: http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)","attempt":3,"retrying_in":120,"elapsed":180.00175752,"max_duration":2592000}
{"level":"info","ts":1748903830.576982,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"caddy-server"}
{"level":"error","ts":1748903830.5773063,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"caddy-server","issuer":"172.12.12.100:8200-v1-pki_int-acme-directory","error":"http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)"}
{"level":"error","ts":1748903830.5773318,"logger":"tls.obtain","msg":"will retry","error":"[caddy-server] Obtain: http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)","attempt":4,"retrying_in":300,"elapsed":300.002753975,"max_duration":2592000}
{"level":"info","ts":1748904130.5776265,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"caddy-server"}
{"level":"error","ts":1748904130.5778828,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"caddy-server","issuer":"172.12.12.100:8200-v1-pki_int-acme-directory","error":"http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)"}
{"level":"error","ts":1748904130.5779035,"logger":"tls.obtain","msg":"will retry","error":"[caddy-server] Obtain: http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)","attempt":5,"retrying_in":600,"elapsed":600.003326158,"max_duration":2592000}

The connection from Caddy to the ACME server is not itself secured with HTTPS, and the Caddy server doesn’t have the Root CA Cert file.

I tried updating the caddyfile to read:

cat > "$HC_LEARN_LAB"/caddy_config/Caddyfile << EOF
   {
      debug
      acme_ca http://172.12.12.100:8200/v1/pki_int/acme/directory
      ca_root /etc/caddy/root_2023_ca.crt
   }

   caddy-server {
      root * /usr/share/caddy
      file_server browse
   }

EOF

With an appropriate update to the docker run command for caddy:

docker run \
    --name caddy-server \
    --hostname caddy-server \
    --network learn-vault \
    --ip 172.12.12.200 \
    --publish 443:443 \
    --volume "$HC_LEARN_LAB"/caddy_config/Caddyfile:/etc/caddy/Caddyfile \
    --volume "$HC_LEARN_LAB"/pki/root_2023_ca.crt:/etc/caddy/root_2023_ca.crt \
    --volume "$HC_LEARN_LAB"/index.html:/usr/share/caddy/index.html \
    --volume "$HC_LEARN_LAB"/caddy_data:/data \
    --detach \
    --rm \
    caddy:2.6.4

And this doesn’t seem to have made any difference. The caddy server still can’t get an HTTPS connection to the ACME server, even though it should now have the Root CA Cert file loaded.

But I’m still getting the same kinds of errors when I restart caddy:

docker logs caddy-server
{"level":"info","ts":1748905202.775071,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1748905202.7757702,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
{"level":"info","ts":1748905202.7762945,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1748905202.7767463,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1748905202.7767599,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1748905202.7768397,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004512d0"}
{"level":"info","ts":1748905202.7768936,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1748905202.7769096,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1748905202.7769375,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1748905202.7769964,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
{"level":"info","ts":1748905202.7770402,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1748905202.777078,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1748905202.7778258,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["caddy-server"]}
{"level":"info","ts":1748905202.7794955,"logger":"tls.obtain","msg":"acquiring lock","identifier":"caddy-server"}
{"level":"info","ts":1748905202.7827442,"logger":"tls.obtain","msg":"lock acquired","identifier":"caddy-server"}
{"level":"info","ts":1748905202.7828298,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"caddy-server"}
{"level":"error","ts":1748905202.7831612,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"caddy-server","issuer":"172.12.12.100:8200-v1-pki_int-acme-directory","error":"http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)"}
{"level":"error","ts":1748905202.7831867,"logger":"tls.obtain","msg":"will retry","error":"[caddy-server] Obtain: http://172.12.12.100:8200/v1/pki_int/acme/directory: insecure CA URL (HTTPS required)","attempt":1,"retrying_in":60,"elapsed":0.000428496,"max_duration":2592000}
{"level":"info","ts":1748905202.7840567,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1748905202.7840729,"msg":"serving initial configuration"}

So, I’m stumped. It looks like this tutorial is a couple of years out of date, and needs to be fixed. Does anyone else have any other suggestions?

Thanks!

2

I can test the tutorial, but outside of that is there some other question/use case you need help with? Just want to make sure I get you unblocked while I test that tutorial.

3

I tested the embedded tutorial using the Launch Terminal banner (just below the version info Manage certificates with ACME clients and the PKI secrets engine | Vault | HashiCorp Developer) and everything worked even with v2.10 of Caddy. I will still test this locally but just wanted to give you an update.

4

I tested this locally on macOS with Apple Silicon and everything worked as expected.

I would back track a couple of steps and ensure all the previous commands (like the PKI setup script) did not throw an error.

jonathanfrappier@Mac pki % docker run \
    --name caddy-server \
    --hostname caddy-server \
    --network learn-vault \
    --ip 10.1.1.200 \
    --publish 443:443 \
    --volume "$HC_LEARN_LAB"/caddy_config/Caddyfile:/etc/caddy/Caddyfile \
    --volume "$HC_LEARN_LAB"/index.html:/usr/share/caddy/index.html \
    --volume "$HC_LEARN_LAB"/caddy_data:/data \
    --detach \
    --rm \
    caddy:2.6.4

7b8d84ee1cad89c3ccc41f05707d244c73087aa89666265739cf63e581b5920f
jonathanfrappier@Mac pki % curl \
    --cacert "$HC_LEARN_LAB"/pki/root_2023_ca.crt \
    --resolve caddy-server:443:127.0.0.1 \
    https://caddy-server

hello world
5

I figured out what the problem was. I had Hashicorp Vault and Caddy running on the same server, and both trying to listen to port 80. That’s not going to work. I found this out the hard way when I punted and tried to use Certbot in -standalone mode, and it ran into the same problem.

I still think there is weirdness with Caddy requiring HTTPS connections to the ACME server, but we have been told to cease and desist any further work with Hashicorp Vault for this purpose. So, I will never get the chance to try with a second server.

6

Ahh yea - port fighting is almost as bad as DNS :slight_smile: Good luck with your project.