Rotating Nomad TLS Certificates

Nomad
1

Hi

Consul has a guide on how to rotate consul TLS certificates. I looked but couldn’t find if there is a similar guide for rotating nomad TLS certificates. More specifically, I am interested in rotating expired certificates.

2

Hi @vkatsikaros,

Unfortunately Nomad doesn’t have such as knowledge base article at the moment. The linked Consul KB can be used as a reference though as Nomad’s implementation is similar.

When an agent’s TLS certificate has expired, they should be replaced, and then the agent process restarted. In the event all certificates in the cluster are expired you should follow the process detailed in the link document. I hope this helps.

Thanks,
jrasell and the Nomad team

3

Thanks for the reply @jrasell

The consul article prescribes a specific order

  1. non leader server agents
  2. leader server agents
  3. client server agents

The problem is I don’t know how to identify the leader server from the nomad cli, since the certificates have expired. Do you know if there is another artifact that could help me identify the leader server?

I tried http api but I couldn’t make it respond

$ curl http://localhost:4646/v1/nodes
Client sent an HTTP request to an HTTPS server.

# ok let's use https
$ curl https://localhost:4646/v1/nodes 
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

# ok let's ignore the certificate
$ curl --insecure https://localhost:4646/v1/nodes 
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0


# not sure if this last one means the nomad http api responds but the response is empty, or there is no response at all
$ curl -v --insecure https://localhost:4646/v1/nodes 
*   Trying 127.0.0.1:4646...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4646 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=server.us-west-2.consul
*  start date: Feb 18 19:08:00 2022 GMT
*  expire date: Feb 18 19:08:00 2023 GMT
*  issuer: C=US; ST=CA; L=San Francisco; street=101 Second Street; postalCode=94105; O=HashiCorp Inc.; CN=Consul Agent CA 315303880351045785287045568140342414353
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /v1/nodes HTTP/1.1
> Host: localhost:4646
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS alert, bad certificate (554):
* OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0