Failed to verify certificate: x509: certificate specifies an incompatible key usage

I am trying to install Consul with my private PKI.
It seems that consul does not like my server cert despite it works fine with Tomcat, LDAP server, etc.

This is the relevant config:

  "tls": {
    "defaults": {
      "key_file": "/tmp/consul.hello.com.plain-key",
      "cert_file": "/tmp/consul.hello.com.crt",
      "ca_file": "/tmp/ca.crt",
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": false
    }
  }

And this is the error I get:
agent.server.rpc: failed to read byte: conn=from=127.0.0.1:53133 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"

Unfortunately there is nothing concrate in the log about the real reason.

CA cert:

$ openssl x509 -text -noout -in /tmp/ca.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            37:9b:62:b5:e2:83:b2:cf:31:27:16:60:83:76:1a:a6:12:56:20:9b
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = hello.com
        Validity
            Not Before: Feb 10 00:00:38 2024 GMT
            Not After : Feb  7 00:00:38 2034 GMT
        Subject: CN = hello.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:f0:1e:cf:a6:1b:48:55:de:34:a9:4a:80:c5:8b:
                    2c:b5:a0:be:04:50:e8:0d:71:fa:c8:c6:54:9b:3d:
                    06:9a:4d:11:96:10:db:6d:ac:e5:05:15:fd:4e:83:
                    11:ae:07:2b:69:43:ee:b4:a7:3a:87:47:76:cb:6a:
                    bc:9c:86:ae:2c:4a:fa:39:9d:3b:ba:1f:59:11:44:
                    49:84:30:6e:f6:d2:d9:94:6b:89:3c:c8:0c:2b:c4:
                    36:b4:4b:8f:4c:01:9a
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Subject Key Identifier: 
                DB:6E:71:3B:A6:F1:69:3E:A5:31:EB:39:E9:BF:F6:8C:C7:D2:FD:81
            X509v3 Authority Key Identifier: 
                keyid:DB:6E:71:3B:A6:F1:69:3E:A5:31:EB:39:E9:BF:F6:8C:C7:D2:FD:81
                DirName:/CN=hello.com
                serial:37:9B:62:B5:E2:83:B2:CF:31:27:16:60:83:76:1A:A6:12:56:20:9B
            X509v3 Key Usage: 
                Certificate Sign, CRL Sign
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:64:02:30:24:97:21:c9:2b:55:a9:6c:b6:23:55:72:3d:44:
        80:21:a8:8a:96:1c:fd:a3:d2:ce:a6:7d:14:4a:49:b8:45:85:
        29:e4:80:24:30:c1:67:ee:f3:13:26:36:e6:2f:db:28:02:30:
        32:fb:05:b5:b5:75:71:4e:2b:82:0b:5e:6c:2d:58:b9:e2:f1:
        13:0a:bc:ec:da:9e:cd:26:79:53:29:27:4b:0d:af:81:d8:9a:
        67:c1:4e:0d:5b:13:2e:4a:a8:74:9b:ae

server cert:

$ openssl x509 -text -noout -in /tmp/consul.hello.com.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ea:92:1f:ba:8c:f8:d0:78:7d:fb:6c:72:93:34:74:ff
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = hello.com
        Validity
            Not Before: Feb 10 00:00:40 2024 GMT
            Not After : May 15 00:00:40 2026 GMT
        Subject: CN = consul.hello.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:71:a6:af:d3:70:7c:58:92:ba:e8:2f:04:25:51:
                    34:8a:18:ab:f5:85:11:15:7e:ef:20:78:17:95:64:
                    71:eb:ed:83:86:b6:8a:0b:23:cf:4d:33:c4:fb:2b:
                    56:df:38:1d:ec:8b:22:c0:bf:22:32:aa:fc:d0:88:
                    a4:f4:ff:40:4c:b8:2b:44:74:31:31:8a:0a:43:58:
                    8a:43:28:66:67:1d:5f:b1:e6:ed:87:18:76:d3:e4:
                    65:13:c5:d3:06:17:48
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Key Identifier: 
                5F:34:D8:0C:09:1D:04:B9:94:73:FA:51:F6:2E:8E:C2:99:D9:0B:8E
            X509v3 Authority Key Identifier: 
                keyid:DB:6E:71:3B:A6:F1:69:3E:A5:31:EB:39:E9:BF:F6:8C:C7:D2:FD:81
                DirName:/CN=hello.com
                serial:37:9B:62:B5:E2:83:B2:CF:31:27:16:60:83:76:1A:A6:12:56:20:9B
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:consul.hello.com
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:65:02:31:00:ea:65:13:52:b5:72:7d:bc:bd:27:b8:ce:92:
        94:73:2e:62:31:c6:cf:93:34:b6:e5:74:17:58:2c:24:c4:95:
        10:82:46:30:d9:7b:a8:50:b0:84:64:1c:59:63:7f:69:48:02:
        30:3a:b2:2a:64:73:b0:15:52:d2:f8:58:95:c7:95:72:2f:96:
        a9:6d:ed:a6:e3:12:bc:bf:86:5c:87:4c:5a:e3:95:e3:80:6f:
        c0:38:e9:7d:e2:27:09:50:3b:d9:f9:40:2e

key:

$ cat /tmp/consul.hello.com.plain-key 
-----BEGIN PRIVATE KEY-----
MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDCf0XlOy7bCWtQRHpQ9
e8j/WMNtIgZsHop97AnXjWJg4UQZugiEKyhw0YGQGJ/cCe2hZANiAARxpq/TcHxY
krroLwQlUTSKGKv1hREVfu8geBeVZHHr7YOGtooLI89NM8T7K1bfOB3siyLAvyIy
qvzQiKT0/0BMuCtEdDExigpDWIpDKGZnHV+x5u2HGHbT5GUTxdMGF0g=
-----END PRIVATE KEY-----

echo (hostname -f)
consul.hello.com

And this is the full error log:

==> Starting Consul agent...
               Version: '1.17.2'
            Build Date: '2024-01-22 16:55:18 +0000 UTC'
               Node ID: '60fd623e-401b-6163-a635-f06e9bc0e833'
             Node name: 'agent-one'
            Datacenter: 'dc1' (Segment: '<all>')
                Server: true (Bootstrap: true)
           Client Addr: [0.0.0.0] (HTTP: -1, HTTPS: 8501, gRPC: -1, gRPC-TLS: 8503, DNS: 8600)
          Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
     Gossip Encryption: true
      Auto-Encrypt-TLS: false
           ACL Enabled: false
     Reporting Enabled: false
    ACL Default Policy: allow
             HTTPS TLS: Verify Incoming: true, Verify Outgoing: true, Min Version: TLSv1_2
              gRPC TLS: Verify Incoming: true, Min Version: TLSv1_2
      Internal RPC TLS: Verify Incoming: true, Verify Outgoing: true (Verify Hostname: false), Min Version: TLSv1_2

==> Log data will now stream in as it occurs:

2024-02-10T00:46:23.981Z [WARN]  agent: bootstrap = true: do not enable unless necessary
2024-02-10T00:46:24.047Z [DEBUG] agent.grpc.balancer: switching server: target=consul://dc1.60fd623e-401b-6163-a635-f06e9bc0e833/server.dc1 from=<none> to=<none>
2024-02-10T00:46:24.063Z [WARN]  agent.auto_config: bootstrap = true: do not enable unless necessary
2024-02-10T00:46:24.106Z [INFO]  agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:60fd623e-401b-6163-a635-f06e9bc0e833 Address:127.0.0.1:8300}]"
2024-02-10T00:46:24.106Z [INFO]  agent.server.raft: entering follower state: follower="Node at 127.0.0.1:8300 [Follower]" leader-address= leader-id=
2024-02-10T00:46:24.108Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: agent-one.dc1 127.0.0.1
2024-02-10T00:46:24.109Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: agent-one 127.0.0.1
2024-02-10T00:46:24.109Z [INFO]  agent.router: Initializing LAN area manager
2024-02-10T00:46:24.110Z [DEBUG] agent.grpc.balancer: switching server: target=consul://dc1.60fd623e-401b-6163-a635-f06e9bc0e833/server.dc1 from=<none> to=dc1-127.0.0.1:8300
2024-02-10T00:46:24.110Z [WARN]  agent.server.serf.wan: serf: Failed to re-join any previously known node
2024-02-10T00:46:24.110Z [WARN]  agent.server.serf.lan: serf: Failed to re-join any previously known node
2024-02-10T00:46:24.111Z [INFO]  agent.server: Adding LAN server: server="agent-one (Addr: tcp/127.0.0.1:8300) (DC: dc1)"
2024-02-10T00:46:24.112Z [INFO]  agent.server: Handled event for server in area: event=member-join server=agent-one.dc1 area=wan
2024-02-10T00:46:24.113Z [INFO]  agent.server.autopilot: reconciliation now disabled
2024-02-10T00:46:24.162Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:37471 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:24.162Z [WARN]  agent: [core][Channel #1 SubChannel #5] grpc: addrConn.createTransport failed to connect to {Addr: "dc1-127.0.0.1:8300", ServerName: "agent-one", }. Err: connection error: desc = "error reading server preface: remote error: tls: bad certificate"
2024-02-10T00:46:24.163Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:44687 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:24.163Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:24.163Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:25.114Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:25.114Z [ERROR] agent.server.autopilot: Error when computing next state: error="context deadline exceeded"
2024-02-10T00:46:25.114Z [DEBUG] agent.server.autopilot: autopilot is now running
2024-02-10T00:46:25.114Z [DEBUG] agent.server.autopilot: state update routine is now running
2024-02-10T00:46:25.114Z [INFO]  agent.server.cert-manager: initialized server certificate management
2024-02-10T00:46:25.114Z [DEBUG] agent.hcp_manager: HCP manager starting
2024-02-10T00:46:25.115Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
2024-02-10T00:46:25.115Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2024-02-10T00:46:25.117Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/tcproute/
2024-02-10T00:46:25.117Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/destinationpolicy/
2024-02-10T00:46:25.117Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/healthstatus/
2024-02-10T00:46:25.117Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/proxystatetemplate/
2024-02-10T00:46:25.117Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/album/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v2/album/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/failoverpolicy/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/workloadidentity/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/executive/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/proxyconfiguration/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedproxyconfiguration/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/service/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/trafficpermissions/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/artist/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/httproute/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/grpcroute/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v2/artist/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/tenancy/v1alpha1/namespace/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/concept/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/destinations/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/serviceendpoints/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/internal/v1/tombstone/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/workload/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/node/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/computedtrafficpermissions/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedexplicitdestinations/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedroutes/
2024-02-10T00:46:25.118Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/recordlabel/
2024-02-10T00:46:25.128Z [INFO]  agent: Starting server: address=[::]:8501 network=tcp protocol=https
2024-02-10T00:46:25.144Z [INFO]  agent: Started gRPC listeners: port_name=grpc_tls address=[::]:8503 network=tcp
2024-02-10T00:46:25.146Z [INFO]  agent: started state syncer
2024-02-10T00:46:25.146Z [INFO]  agent: Consul agent running!
2024-02-10T00:46:26.115Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
2024-02-10T00:46:27.115Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
2024-02-10T00:46:27.182Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:57581 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:27.182Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:27.182Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:28.116Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
2024-02-10T00:46:28.116Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:29.116Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
2024-02-10T00:46:29.179Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:49559 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:29.179Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:29.180Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:30.116Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:30.117Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
2024-02-10T00:46:30.915Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-02-10T00:46:30.915Z [INFO]  agent.server.raft: entering candidate state: node="Node at 127.0.0.1:8300 [Candidate]" term=3
2024-02-10T00:46:30.917Z [DEBUG] agent.server.raft: voting for self: term=3 id=60fd623e-401b-6163-a635-f06e9bc0e833
2024-02-10T00:46:30.920Z [DEBUG] agent.server.raft: calculated votes needed: needed=1 term=3
2024-02-10T00:46:30.920Z [DEBUG] agent.server.raft: vote granted: from=60fd623e-401b-6163-a635-f06e9bc0e833 term=3 tally=1
2024-02-10T00:46:30.920Z [INFO]  agent.server.raft: election won: term=3 tally=1
2024-02-10T00:46:30.920Z [INFO]  agent.server.raft: entering leader state: leader="Node at 127.0.0.1:8300 [Leader]"
2024-02-10T00:46:30.920Z [DEBUG] agent.hcp_manager: HCP triggering status update
2024-02-10T00:46:30.920Z [DEBUG] agent.controller-runtime: controller running: managed_type=internal.v1.Tombstone
2024-02-10T00:46:30.920Z [INFO]  agent.server: cluster leadership acquired
2024-02-10T00:46:30.920Z [INFO]  agent.server: New leader elected: payload=agent-one
2024-02-10T00:46:30.927Z [DEBUG] agent.server.xds_capacity_controller: updating drain rate limit: rate_limit=1
2024-02-10T00:46:30.928Z [INFO]  agent.server.autopilot: reconciliation now enabled
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="federation state anti-entropy"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="federation state pruning"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="streaming peering resources"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="metrics for streaming peering resources"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="peering deferred deletion"
2024-02-10T00:46:30.928Z [INFO]  connect.ca: initialized primary datacenter CA from existing CARoot with provider: provider=consul
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="intermediate cert renew watch"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="CA root pruning"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="CA root expiration metric"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="CA signing expiration metric"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="virtual IP version check"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: started routine: routine="config entry controllers"
2024-02-10T00:46:30.928Z [DEBUG] agent.server: successfully established leadership: duration="562.613µs"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: stopping routine: routine="virtual IP version check"
2024-02-10T00:46:30.928Z [INFO]  agent.leader: stopped routine: routine="virtual IP version check"
2024-02-10T00:46:31.118Z [DEBUG] agent.server.cert-manager: CA config watch fired - updating auto TLS server name: name=server.dc1.peering.80d89f87-45b5-e936-4908-735fd86f8fd0.consul
2024-02-10T00:46:31.148Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:44077 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:31.151Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:31.151Z [ERROR] agent: yamux: Failed to write header: write tcp 127.0.0.1:44077->127.0.0.1:8300: write: broken pipe
2024-02-10T00:46:31.189Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:53683 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:31.189Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:31.189Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:32.115Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:33.178Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:46873 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:33.178Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:33.178Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:33.893Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2024-02-10T00:46:33.897Z [INFO]  agent: Synced node info
2024-02-10T00:46:34.115Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:34.265Z [DEBUG] agent.server.cert-manager: got cache update event: correlationID=leaf error=<nil>
2024-02-10T00:46:34.265Z [DEBUG] agent.server.cert-manager: leaf certificate watch fired - updating auto TLS certificate: uri=spiffe://80d89f87-45b5-e936-4908-735fd86f8fd0.consul/agent/server/dc/dc1
2024-02-10T00:46:35.144Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:51601 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:35.146Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:35.146Z [ERROR] agent: yamux: Failed to write header: write tcp 127.0.0.1:51601->127.0.0.1:8300: write: broken pipe
2024-02-10T00:46:35.179Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:51953 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:35.179Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:35.179Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:36.115Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:36.592Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2024-02-10T00:46:36.592Z [DEBUG] agent: Node info in sync
2024-02-10T00:46:36.592Z [DEBUG] agent: Node info in sync
2024-02-10T00:46:37.151Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:57325 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:37.155Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:37.155Z [ERROR] agent: yamux: Failed to write header: write tcp 127.0.0.1:57325->127.0.0.1:8300: write: broken pipe
2024-02-10T00:46:37.193Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:43383 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:37.193Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:37.194Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:38.115Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:39.212Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:34501 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:39.215Z [ERROR] agent: yamux: Failed to write header: write tcp 127.0.0.1:34501->127.0.0.1:8300: write: broken pipe
2024-02-10T00:46:39.215Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:39.281Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:53495 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:39.281Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:39.282Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:40.122Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:41.153Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:34757 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:41.158Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:41.158Z [ERROR] agent: yamux: Failed to write header: write tcp 127.0.0.1:34757->127.0.0.1:8300: use of closed network connection
2024-02-10T00:46:41.205Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:49175 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:41.205Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:41.205Z [WARN]  agent: error getting server health from server: server=agent-one error="rpc error making call: EOF"
2024-02-10T00:46:42.116Z [WARN]  agent: error getting server health from server: server=agent-one error="context deadline exceeded"
2024-02-10T00:46:43.147Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:48423 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"
2024-02-10T00:46:43.149Z [ERROR] agent: yamux: Failed to read header: remote error: tls: bad certificate
2024-02-10T00:46:43.149Z [ERROR] agent: yamux: Failed to write header: write tcp 127.0.0.1:48423->127.0.0.1:8300: use of closed network connection
2024-02-10T00:46:43.190Z [ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.1:51819 error="tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"

I guess that Consul has a problem with my consul.hello.com.crt file. But what is wrong with it? Could you please help?

Hi @zappee,

If you look at the Extended Key Usage of your server certificate, you will see that it only has "TLS Web Server Authentication". Your certificate should have "TLS Web Server Authentication" and “TLS Web Client Authentication” in the Extended Key Usage.

       X509v3 Extended Key Usage: 
           TLS Web Server Authentication

Since every agent acts as a server and client, and when configured to do mTLS (using verify_incoming and verify_outgoing), server and client authentication constraints should be there or left empty.

You can compare this with a cert generated using the consul tls CLI.

X509v3 Extended Key Usage:
    TLS Web Server Authentication, TLS Web Client Authentication

I hope this helps!

That was the issue. Thanks for your help.

I added the following line to the easy-rsa X509 extensions file and that solved this:
extendedKeyUsage = serverAuth,clientAuth


Unfortunately, I have a 3rd cert related issue but this is a new storry.
Using the HTTP the console appears, but if I use HTTPS I get a SEC_ERROR_BAD_SIGNATURE.

$ curl http://localhost:8500/ui/

<!DOCTYPE html>
<!--
 Copyright (c) HashiCorp, Inc.
 SPDX-License-Identifier: BUSL-1.1
-->

<html lang="en" class="ember-loading">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>Consul by HashiCorp</title>
    ...

But:

$ curl https://localhost:8501/ui/ --insecure
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.12: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

I need to check what is this issue again.

Hi @zappee,

This is because you have tls.defaults.verify_incoming set to true, which applies to the HTTPS port as well. This means the HTTPs interface expects you to do client cert authentication.

If you only want to use one-way TLS, set the tls.https.verify_incoming to false and restart the agent.

ref: https://developer.hashicorp.com/consul/docs/agent/config/config-files#tls_https_verify_incoming

1 Like

Thanks a lot. Now (I hope :slight_smile:) it is clear how Consul operates with certs and what mTSL means in this context.

I generated a new “fake” client cert for testing purposes, and used it with curl. This way the HTTPS GET works like a charm with verify_incoming=true.

curl --cert aaa.hello.com.crt --key aaa.hello.com.key --cacert ca.crt https://localhost:8501/ui/

As I see/guess the verify_incoming=false can be used on consul server which provides the consule web console as well. Then verify_incoming=true can be used on consul clients that I am planning to put in different docker containers where my rest endpoints (java spring-boot ) will run.

I hope that consul will not have an issue if I install consul clients into the docker containers that provide the rest services. That way the rest applications in the container can communicate with consul client using localhost and consul client will connect to consul server via the network. I will have ~100 service containers which mean 100 consule clients.

Thanks for the help.

H @zappee,

Your understanding of verify_incoming and certs is correct. Regarding your use case, if you are interested, I recommend you explore HashiCorp Nomad to provision your containers and connect them. Nomad integrates with Consul, and you will only have to run one Consul Client per container host.

Ref:

Considering all the queries on this topic have been answered, I recommend creating new topics if you have follow-up questions.