Consul TLS verification from k8s cluster to Consul Server

[Lord-Y]

Hello guys,

I have exactly the same pb. I have 3 VMs for consul servers, 1 consul client in a VM and a kubernetes cluster in which there is 3 clients. All clients are able to connect to consul servers except connect inject who is throwing msg “consul-connect-injector-webhook-deployment-6c75d8b79d-m8lck get-auto-encrypt-client-ca 2020-09-01T07:19:00.278Z [ERROR] Error retrieving CA roots from Consul: err=“Get “https://XXXXX:8501/v1/agent/connect/ca/roots”: remote error: tls: bad certificate””.

My Ca setup is:
consul tls ca create -name-constraint
-additional-name-constraint={EXTERNAL_DOMAIN} \ -additional-name-constraint={FQDN_OF_CONSUL_SERVER01}
-additional-name-constraint={FQDN_OF_CONSUL_SERVER02} \ -additional-name-constraint={FQDN_OF_CONSUL_SERVER03}

consul tls cert create -server -dc euw1 -additional-dnsname={EXTERNAL_DOMAIN} -additional-dnsname={FQDN_OF_CONSUL_SERVER01}
consul tls cert create -server -dc euw1 -additional-dnsname={EXTERNAL_DOMAIN} -additional-dnsname={FQDN_OF_CONSUL_SERVER02}
consul tls cert create -server -dc euw1 -additional-dnsname={EXTERNAL_DOMAIN} -additional-dnsname={FQDN_OF_CONSUL_SERVER03}

Here is the config of my 1st consul server (almost the same for the 2 others):
{
“acl”: {
“default_policy”: “allow”,
“enable_token_persistence”: true,
“enabled”: true
},
“advertise_addr”: “IP_OF_SERVER01”,
“auto_encrypt”: {
“allow_tls”: true
},
“bind_addr”: “0.0.0.0”,
“bootstrap_expect”: 3,
“ca_file”: “/etc/consul.d/euw1-consul-agent-ca.pem”,
“cert_file”: “/etc/consul.d/euw1-server-consul.pem”,
“client_addr”: “0.0.0.0”,
“connect”: {
“enabled”: true
},
“data_dir”: “/data/consul”,
“datacenter”: “euw1”,
“encrypt”: “XXXXX”,
“key_file”: “/etc/consul.d/euw1-server-consul-key.pem”,
“log_level”: “DEBUG”,
“node_name”: “${FQDN_OF_CONSUL_SERVER01}”,
“performance”: {
“raft_multiplier”: 1
},
“ports”: {
“grpc”: 8502,
“https”: 8501
},
“retry_join”: [
“IP_OF_SERVER01”,
“IP_OF_SERVER02”,
“IP_OF_SERVER03”
],
“server”: true,
“ui”: true,
“verify_incoming”: true,
“verify_outgoing”: true,
“verify_server_hostname”: true
}

So in order to verify if the certificates are good or not, I performed my curl command:

curl -vvvk --cert euw1-consul-agent-ca.pem --key /root/consul-agent-ca-key.pem https://XXXXX:8501/v1/agent/connect/ca/roots --cacert euw1-consul-agent-ca.pem

And the certificate is verified as I have the message “SSL certificate verify ok”:

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=server.euw1.consul
*  start date: Sep  1 06:26:40 2020 GMT
*  expire date: Jun 17 07:26:40 2294 GMT
*  issuer: C=US; ST=CA; L=San Francisco; street=101 Second Street; postalCode=94105; O=HashiCorp Inc.; CN=Consul Agent CA 251292510108551272556755631721680811671
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5649639eef50)
> GET /v1/agent/connect/ca/roots HTTP/2
> Host: XXXXX:8501
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< age: 0
< content-type: application/json
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-index: 3828
< x-consul-knownleader: false
< x-consul-lastcontact: 0
< content-length: 1435
< date: Tue, 01 Sep 2020 06:45:31 GMT

So without TLS Handshake issues, I don’t understand why connect-inject is not able to verify the servers.
I check the secrets for the tls.crt and tls.key and they are the right ones.

Following the same issue here Remote error: tls: bad certificate for K8S consul clients I tried to make the same with ACLs and the issue was the same “tls bad certificate”.

My helm values.yaml are:

global:
  enabled: false
  name: consul
  domain: consul
  image: "consul:1.8.3"
  imagePullSecrets: []
  imageK8S: "hashicorp/consul-k8s:0.18.1"
  imageEnvoy: "envoyproxy/envoy-alpine:v1.14.2"
  datacenter: euw1
  enablePodSecurityPolicies: false
  gossipEncryption:
    secretName: "consul-secrets"
    secretKey: "gossipEncryptionKey"
  tls:
    enabled: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: []
    serverAdditionalIPSANs: []
    verify: true
    httpsOnly: true
    caCert:
      secretName: "consul-secrets"
      secretKey: tls.crt
    caKey:
      secretName: "consul-secrets"
      secretKey: tls.key
  enableConsulNamespaces: false
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-secrets
      secretKey: token_mgmt
    createReplicationToken: false
    replicationToken:
      secretName: null
      secretKey: null
  federation:
    enabled: false
    createFederationSecret: false
  lifecycleSidecarContainer:
    resources:
      requests:
        memory: "25Mi"
        cpu: "20m"
      limits:
        memory: "50Mi"
        cpu: "20m"
server:
  enabled: "-"
  image: null
  replicas: 3
  enterpriseLicense:
    secretName: null
    secretKey: null
  storage: 10Gi
  storageClass: null
  connect: true
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "100Mi"
      cpu: "100m"
  updatePartition: 0
  disruptionBudget:
    enabled: true
    maxUnavailable: null
  extraConfig: |
    {}
  extraVolumes: []
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "consul.name" . }}
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname
  tolerations: ""
  nodeSelector: null
  priorityClassName: ""
  extraLabels: null
  annotations: null
  service:
    annotations: null
  extraEnvironmentVars: {}
  disableFsGroupSecurityContext : false
externalServers:
  enabled: true
  hosts:
  - [REDACTED]
  httpsPort: 8501
  tlsServerName: null
  useSystemRoots: false
  k8sAuthMethodHost: https://172.16.0.82
client:
  enabled: true
  image: null
  join:
  - [REDACTED]
  dataDirectoryHostPath: null
  grpc: true
  exposeGossipPorts: true
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "100Mi"
      cpu: "100m"
  extraConfig: |
    {
      "auto_encrypt": {
        "tls": true
      }
    }
  extraVolumes:
  - type: secret
    name: consul-secrets
  tolerations: ""
  nodeSelector: null
  affinity: |
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/master
            operator: DoesNotExist
  priorityClassName: "consul"
  annotations: null
  extraEnvironmentVars: {}
  dnsPolicy: null
  hostNetwork: false
  updateStrategy: |
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  snapshotAgent:
    enabled: false
    replicas: 2
    configSecret:
      secretName: null
      secretKey: null
    resources:
      requests:
        memory: "50Mi"
        cpu: "50m"
      limits:
        memory: "50Mi"
        cpu: "50m"
    caCert: null
dns:
  enabled: "-"
  clusterIP: null
  annotations: null
ui:
  enabled: "-"
  service:
    enabled: true
    type: null
    annotations: null
    additionalSpec: null
syncCatalog:
  enabled: false
  image: null
  toConsul: true
  toK8S: true
  k8sPrefix: null
  k8sAllowNamespaces: ["*"]
  k8sDenyNamespaces: ["kube-system", "kube-public"]
  k8sSourceNamespace: null
  consulNamespaces:
    consulDestinationNamespace: "default"
    mirroringK8S: false
    mirroringK8SPrefix: ""
  addK8SNamespaceSuffix: true
  consulPrefix: null
  k8sTag: null
  syncClusterIPServices: true
  nodePortSyncType: ExternalFirst
  aclSyncToken:
    secretName: null
    secretKey: null
  nodeSelector: null
  affinity: null
  tolerations: null
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m"
    limits:
      memory: "50Mi"
      cpu: "50m"
  logLevel: info
  consulWriteInterval: null
connectInject:
  enabled: true
  imageConsul: null
  resources:
    requests:
      memory: "50Mi"
      cpu: "50m"
    limits:
      memory: "50Mi"
      cpu: "50m"
  imageEnvoy: null
  namespaceSelector: null
  k8sAllowNamespaces: ["*"]
  k8sDenyNamespaces: []
  consulNamespaces:
    consulDestinationNamespace: "default"
    mirroringK8S: false
    mirroringK8SPrefix: ""
  certs:
    secretName: null
    caBundle: ""
    certName: tls.crt
    keyName: tls.key
  nodeSelector: null
  affinity: null
  tolerations: null
  aclBindingRuleSelector: "serviceaccount.name!=default"
  overrideAuthMethodName: ""
  aclInjectToken:
    secretName: null
    secretKey: null
  centralConfig:
    enabled: true
    defaultProtocol: null
    proxyDefaults: |
      {}
  sidecarProxy:
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 100Mi
        cpu: 100m
  initContainer:
    resources:
      requests:
        memory: "25Mi"
        cpu: "50m"
      limits:
        memory: "150Mi"
        cpu: "50m"
meshGateway:
  enabled: false
  globalMode: local
  replicas: 2
  wanAddress:
    source: "Service"
    port: 443
    static: ""
  service:
    enabled: true
    type: LoadBalancer
    port: 443
    nodePort: null
    annotations: null
    additionalSpec: null
  imageEnvoy: envoyproxy/envoy-alpine:v1.14.2
  hostNetwork: false
  dnsPolicy: null
  consulServiceName: "mesh-gateway"
  containerPort: 8443
  hostPort: null
  resources:
    requests:
      memory: "100Mi"
      cpu: "100m"
    limits:
      memory: "100Mi"
      cpu: "100m"
  initCopyConsulContainer:
    resources:
      requests:
        memory: "25Mi"
        cpu: "50m"
      limits:
        memory: "150Mi"
        cpu: "50m"
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "consul.name" . }}
              release: "{{ .Release.Name }}"
              component: mesh-gateway
          topologyKey: kubernetes.io/hostname
  tolerations: null
  nodeSelector: null
  priorityClassName: ""
  annotations: null
ingressGateways:
  enabled: false
  defaults:
    replicas: 2
    service:
      type: ClusterIP
      ports:
        - port: 8080
          nodePort: null
        - port: 8443
          nodePort: null
      annotations: null
      additionalSpec: null
    resources:
      requests:
        memory: "100Mi"
        cpu: "100m"
      limits:
        memory: "100Mi"
        cpu: "100m"
    initCopyConsulContainer:
      resources:
        requests:
          memory: "25Mi"
          cpu: "50m"
        limits:
          memory: "150Mi"
          cpu: "50m"
    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "consul.name" . }}
                release: "{{ .Release.Name }}"
                component: ingress-gateway
            topologyKey: kubernetes.io/hostname
    tolerations: null
    nodeSelector: null
    priorityClassName: ""
    annotations: null
    consulNamespace: "default"
  gateways:
    - name: ingress-gateway
terminatingGateways:
  enabled: false
  defaults:
    replicas: 2
    extraVolumes: []
    resources:
      requests:
        memory: "100Mi"
        cpu: "100m"
      limits:
        memory: "100Mi"
        cpu: "100m"
    initCopyConsulContainer:
      resources:
        requests:
          memory: "25Mi"
          cpu: "50m"
        limits:
          memory: "150Mi"
          cpu: "50m"
    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "consul.name" . }}
                release: "{{ .Release.Name }}"
                component: terminating-gateway
            topologyKey: kubernetes.io/hostname
    tolerations: null
    nodeSelector: null
    priorityClassName: ""
    annotations: null
    consulNamespace: "default"
  gateways:
    - name: terminating-gateway
tests:
  enabled: true

So are you guys were able to make this kind of setup correctly? @ashwin-venkatesh @ishustava @lkysow any help please?