# Remote error: tls: bad certificate for K8S consul clients

**URL:** <https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422>\
**Category:** Consul\
**Tags:** k8s\
**Created:** [August 5, 2020, 10:38am UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422 "2020-08-05T10:38:48Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 5, 2020, 10:38am UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/1 "2020-08-05T10:38:48Z")

</div>

Hi,  
I’m trying to setup consul clients on my AKS cluster (consul server is outside the K8S), but deployment is failing with the following error on consul-server-acl-init job pod:

```auto
Failure: calling /agent/self to get datacenter: err="Get "https://10.162.34.220:8501/v1/agent/self": remote error: tls: bad certificate"

```

I double checked the ca certificate provided as K8S secret and it looks OK. It’s correctly mapped to pod and contains correct data. Same certificate is used by other consul client deployed outside the kubernetes and everything is OK.

Here is my consul helm config file:

```auto
global:
  enabled: false
  name: consul
  datacenter: consul-azure-dc
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-acl-token  
      secretKey: bootstrap-token  
  gossipEncryption:
    secretName: consul-gossip-encryption
    secretKey: gossip
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts: ["10.162.34.220","10.162.34.222","10.162.34.221"]
  k8sAuthMethodHost: https://my-aks-cluster.hcp.westeurope.azmk8s.io:443
  #useSystemRoots: true
client:
  enabled: true
  join: ["10.162.34.220","10.162.34.222","10.162.34.221"]
connectInject:
  enabled: false
  default: false 

```

Maybe I’m doing something wrong? Any suggestions.  
Thanks in advance

---

<div class="post-metadata">

**Author:** ![ishustava](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/ishustava/32/2120_2.png) [@ishustava](https://discuss.hashicorp.com/u/ishustava)\
**Post date:** [August 10, 2020, 6:52pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/2 "2020-08-10T18:52:34Z")

</div>

Hey @andriktr,

Is your server configured to only allow mTLS connections, i.e. does it have `verify_incoming` set to true? Generally, if you have ACLs enabled, mTLS doesn’t provide any extra layers of security, so I’d recommend turning it off and just keeping “regular”, one-way TLS plus ACLs turned on.

---

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 11, 2020, 7:34am UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/3 "2020-08-11T07:34:40Z")

</div>

Hi,  
Thanks for reply. Yes my consul server is configured with `verify_incoming = true`. And normally it should work with such config or at least I think so 🙂 . After setting ` manageSystemACLs: false` my K8S clients joined the cluster however I still can’t enable connectInject because of same reason. Init container named `get-auto-encrypt-client-ca` of consul-connect-injector-webhook pod is not able to get the client CA and failing with same error:

```auto
[ERROR] Error retrieving CA roots from Consul: err="Get "https://consul server:8501/v1/agent/connect/ca/roots": remote error: tls: bad certificate"

```

Inspecting the init container shows that the command is running and produces mentioned error message:

```auto
consul-k8s get-consul-client-ca -output-file=/consul/tls/client/ca/tls.crt -server-addr=consulserver -server-port=8501 -ca-file=/consul/tls/ca/tls.crt

```

I pretty much sure that if I will set `verify_incoming = false` on my consul server this will solve the issue however according the documentation it’s recommended to keep it enabled.

Currently my config yaml looks so:

```auto
global:
  enabled: false
  name: consul
  image: "consul:1.8.1"
  datacenter: consul-azure-dc
  acls:
    manageSystemACLs: false
    bootstrapToken:
      secretName: consul-acl-token  
      secretKey: bootstrap-token  
  gossipEncryption:
    secretName: consul-gossip-encryption
    secretKey: gossip
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt

externalServers:
  enabled: true
  hosts: ["redacted"]
  k8sAuthMethodHost: redacted:443
  useSystemRoots: false

client:
  enabled: true
  join: ["redacted"]
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "100Mi"
      cpu: "100m" 

connectInject:
  enabled: true
  default: false
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m"
    limits:
      memory: "50Mi"
      cpu: "50m"
  centralConfig:
    enabled: false

```

Thanks

---

<div class="post-metadata">

**Author:** ![ishustava](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/ishustava/32/2120_2.png) [@ishustava](https://discuss.hashicorp.com/u/ishustava)\
**Post date:** [August 11, 2020, 5:42pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/4 "2020-08-11T17:42:39Z")

</div>

Hey @andriktr,

It looks like you are getting the same TLS error: `bad certificate`.

When you have [`verify_incoming`](https://www.consul.io/docs/agent/options#verify_incoming) set on the server, it will expect all clients, RPC or HTTP, to present a client certificate and will reject any connections that don’t do so.

In this case, our helm chart does not support servers that require HTTP clients to present client certs. So when some components are talking to the server over HTTPS, they don’t present a certificate, and you’re seeing the `bad certificate` error in your logs.

To fix your problem, I recommend turning ACLs in the helm chart back on and setting `verify_incoming_https` on the server to `false`, but `verify_incoming_rpc` to `true`. That way you can still ensure only trusted Consul clients are joining the cluster, but for HTTPS connections you don’t need that since you’re using ACLs already.

---

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 11, 2020, 7:04pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/5 "2020-08-11T19:04:42Z")

</div>

Hi @ishustava ,  
Thanks for the answer.  
According the helm chart reference in consul documentation `manageSystemACLs: false` should only be enabled if consul servers running inside the kubernetes.

 ![image](https://global.discourse-cdn.com/hashicorp/original/2X/d/d9b96d321074edc50eacfef11f3ad20e863761c7.png)  
In my case consul servers are outside the Kubernetes installed on a separate VM’s.

Also as far as I understand command `consul-k8s get-consul-client-ca -output-file=/consul/tls/client/ca/tls.crt -server-addr=consulserver -server-port=8501 -ca-file=/consul/tls/ca/tls.crt` executed in init container exactly tries to retrieve the client certificate which is expected in case of auto\_encrypt mode.

Just want to clarify one more time that the problem here is not with consul client pods because they are up, running and joined to the cluster. The problem is with consul-connect-injector-webhook pod:

 ![image](https://global.discourse-cdn.com/hashicorp/original/2X/6/6c71f5d0e81ca12becb4524fea1b1da091962fd7.png)

Concluding your answer it’s sounds that it’s not possible to appropriately join K8S consul clients to outside consul cluster and have `verify_incoming` enabled on consul servers together with

```auto
connectInject:
  enabled: true

```

on K8S side

If so IMO this most probably should be mentioned in consul documentation [Secure Consul Agents](https://learn.hashicorp.com/tutorials/consul/tls-encryption-secure) section or in [Consul Servers Outside of Kubernetes](https://www.consul.io/docs/k8s/installation/deployment-configurations/servers-outside-kubernetes) section.

Thank you.

---

<div class="post-metadata">

**Author:** ![ishustava](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/ishustava/32/2120_2.png) [@ishustava](https://discuss.hashicorp.com/u/ishustava)\
**Post date:** [August 11, 2020, 10:02pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/6 "2020-08-11T22:02:33Z")

</div>

Apologies @andriktr, these docs were out of date. It has now been fixed. I think you already saw that we have more docs on how to enable ACLs with external servers [here](https://www.consul.io/docs/k8s/installation/deployment-configurations/servers-outside-kubernetes#configuring-acls).

> Also as far as I understand command `consul-k8s get-consul-client-ca -output-file=/consul/tls/client/ca/tls.crt -server-addr=consulserver -server-port=8501 -ca-file=/consul/tls/ca/tls.crt` executed in init container exactly tries to retrieve the client certificate which is expected in case of auto\_encrypt mode.

That’s not quite true. It’s retrieving the CA that has signed the Consul client’s certificate.

> Concluding your answer it’s sounds that it’s not possible to appropriately join K8S consul clients to outside consul cluster and have `verify_incoming` enabled on consul servers together with

You are correct. I’ll add it to our docs, thanks for pointing this out!

---

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 12, 2020, 5:26am UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/7 "2020-08-12T05:26:41Z")

</div>

Hi again @ishustava thanks for clarification.

> That’s not quite true. It’s retrieving the CA that has signed the Consul client’s certificate.

But the CA’s certificate is already provided in the command in the argument`-ca-file=/consul/tls/ca/tls.crt`

This certificate exist in the container in the mentioned path `/consul/tls/ca/tls.crt` and it comes from Kubernetes secret referenced in helm chart tls configuration:

```auto
tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt

```

What kind of CA it tries to retrieve then?

Also [here](https://discuss.hashicorp.com/t/consul-verified-tls-from-pods-in-kubernetes-cluster/9208/9) some discussions about `consul-k8s get-consul-client-ca`.

Thank you.

---

<div class="post-metadata">

**Author:** ![lkysow](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lkysow/32/1105_2.png) [@lkysow](https://discuss.hashicorp.com/u/lkysow)\
**Post date:** [August 12, 2020, 5:57pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/8 "2020-08-12T17:57:29Z")

</div>

So there’s two CA’s in play. There’s the CA for the servers and then when auto-encrypt is enabled, there’s a **separate** CA for the consul clients.

What we need to do is talk to the server to get the separate client CA. To do that we use the get-consul-client-ca command. However to make the call to the server, we need the server CA. Thats what’s being passed in by `-ca-file=/consul/tls/ca/tls.crt`. Then for subsequent calls to the Consul clients we need to use the client CA.

---

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 12, 2020, 6:23pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/9 "2020-08-12T18:23:28Z")

</div>

@lkysow Thanks for clarification now it’s clear.

The last thing I would like to figure out is the

```auto
  acls:
    manageSystemACLs: false

```

Should I enable it if my consul servers are outside the cluster. Description in the documentation is really confusing me:

 ![image](https://global.discourse-cdn.com/hashicorp/original/2X/5/5cc87f777d8874c397080669e0e3d1fddcc22853.png)

Thanks in advance

---

<div class="post-metadata">

**Author:** ![lkysow](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lkysow/32/1105_2.png) [@lkysow](https://discuss.hashicorp.com/u/lkysow)\
**Post date:** [August 12, 2020, 6:50pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/10 "2020-08-12T18:50:08Z")

</div>

Hi, yes you can enable this if you want ACLs. We have a PR to update the docs but it hasn’t been merged yet.

---

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 13, 2020, 4:11pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/11 "2020-08-13T16:11:49Z")

</div>

@ishustava @lkysow Thanks for the advices, really appreciate it.

---

<div class="post-metadata">

**Author:** ![andriktr](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/andriktr/32/10193_2.png) [@andriktr](https://discuss.hashicorp.com/u/andriktr)\
**Post date:** [August 17, 2020, 12:46pm UTC](https://discuss.hashicorp.com/t/remote-error-tls-bad-certificate-for-k8s-consul-clients/12422/12 "2020-08-17T12:46:24Z")

</div>

Hey @ishustava @lkysow maybe you will also have some insights on my another issue with setting up consul client on K8S. The problem is that if I set **acl default policy to deny** K8S consul client can’t join the cluster. Client container logs show the following:

```auto
==> Starting Consul agent...
           Version: '1.8.3+ent'
           Node ID: '5a64aa6b-178f-f732-d862-fe8482e0fbe6'
         Node name: 'redacted'     
        Datacenter: 'consul-azure-dc' (Segment: '')       
            Server: false (Bootstrap: false)
       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: 8501, gRPC: 8502, DNS: 8600)
      Cluster Addr: redacted (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: true
==> Log data will now stream in as it occurs:
    2020-08-17T07:44:47.304Z [INFO] agent.client.serf.lan: serf: EventMemberJoin: redacted redacted
    2020-08-17T07:44:47.304Z [WARN] agent.client.manager: No servers available
    2020-08-17T07:44:47.514Z [WARN] agent.client: AutoEncrypt failed: error="rpcinsecure error making call: rpcinsecure error making call: ACL not found"
    2020-08-17T07:44:47.522Z [WARN] agent.client: AutoEncrypt failed: error="rpcinsecure error making call: rpcinsecure error making call: ACL not found"
    2020-08-17T07:44:47.595Z [WARN] agent.client: AutoEncrypt failed: error="rpcinsecure error making call: ACL not found"
    2020-08-17T07:44:47.595Z [WARN] agent.client: retrying AutoEncrypt: retry_interval=31.706936702s

```

Here is the my Helm chart config.yaml

```auto
  enabled: false
  name: consul
  image: "hashicorp/consul-enterprise:1.8.3-ent"
  datacenter: consul-azure-dc
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-acl-token  
      secretKey: bootstrap-token  
  gossipEncryption:
    secretName: consul-gossip-encryption
    secretKey: gossip
  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true
    httpsOnly: false
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts: ["redacted"]
  k8sAuthMethodHost: redacted:443
  useSystemRoots: false
client:
  enabled: true
  join: ["redacted"]
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "100Mi"
      cpu: "100m" 
connectInject:
  enabled: true
  default: false
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m"
    limits:
      memory: "50Mi"
      cpu: "50m"
  centralConfig:
    enabled: false
syncCatalog:
  enabled: false
  toConsul: false
  toK8S: false
  default: false 
  k8sPrefix: null
  k8sAllowNamespaces: ["*"]
  k8sDenyNamespaces: ["kube-system", "kube-public", "kube-node-lease", "default", "kubernetes-dashboard", "velero", "kured"]

```

Here is consul.hcl file on the consul server:

```auto
datacenter = "consul-azure-dc"
retry_join = ["redacted"]
retry_join_wan = ["redacted"] 
data_dir = "/opt/consul"
encrypt = "redacted"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/consul-azure-dc-server-consul-0.pem"   
key_file = "/etc/consul.d/consul-azure-dc-server-consul-0-key.pem"
verify_incoming = false
verify_incoming_rpc = true
verify_outgoing = true
verify_server_hostname = true
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens = {
    agent = "redacted"
  }
}
auto_encrypt {
  allow_tls = true
}
performance {
  raft_multiplier = 1
}
ports {
  https = 8501
}

```

Here is the part of the same consul server configuration separated to the server.hcl:

```auto
server = true
bootstrap_expect = 3
ui = true
client_addr = "0.0.0.0"
connect {
  enabled = true
}
primary_datacenter = "consul-azure-dc"

```

Thanks in advance!!!
