Hello guys,
I have exactly the same pb. I have 3 VMs for consul servers, 1 consul client in a VM and a kubernetes cluster in which there is 3 clients. All clients are able to connect to consul servers except connect inject who is throwing msg “consul-connect-injector-webhook-deployment-6c75d8b79d-m8lck get-auto-encrypt-client-ca 2020-09-01T07:19:00.278Z [ERROR] Error retrieving CA roots from Consul: err=“Get “https://XXXXX:8501/v1/agent/connect/ca/roots”: remote error: tls: bad certificate””.
My Ca setup is:
consul tls ca create -name-constraint
-additional-name-constraint={EXTERNAL_DOMAIN} \
-additional-name-constraint={FQDN_OF_CONSUL_SERVER01}
-additional-name-constraint={FQDN_OF_CONSUL_SERVER02} \
-additional-name-constraint={FQDN_OF_CONSUL_SERVER03}
consul tls cert create -server -dc euw1 -additional-dnsname={EXTERNAL_DOMAIN} -additional-dnsname={FQDN_OF_CONSUL_SERVER01}
consul tls cert create -server -dc euw1 -additional-dnsname={EXTERNAL_DOMAIN} -additional-dnsname={FQDN_OF_CONSUL_SERVER02}
consul tls cert create -server -dc euw1 -additional-dnsname={EXTERNAL_DOMAIN} -additional-dnsname={FQDN_OF_CONSUL_SERVER03}
Here is the config of my 1st consul server (almost the same for the 2 others):
{
“acl”: {
“default_policy”: “allow”,
“enable_token_persistence”: true,
“enabled”: true
},
“advertise_addr”: “IP_OF_SERVER01”,
“auto_encrypt”: {
“allow_tls”: true
},
“bind_addr”: “0.0.0.0”,
“bootstrap_expect”: 3,
“ca_file”: “/etc/consul.d/euw1-consul-agent-ca.pem”,
“cert_file”: “/etc/consul.d/euw1-server-consul.pem”,
“client_addr”: “0.0.0.0”,
“connect”: {
“enabled”: true
},
“data_dir”: “/data/consul”,
“datacenter”: “euw1”,
“encrypt”: “XXXXX”,
“key_file”: “/etc/consul.d/euw1-server-consul-key.pem”,
“log_level”: “DEBUG”,
“node_name”: “${FQDN_OF_CONSUL_SERVER01}”,
“performance”: {
“raft_multiplier”: 1
},
“ports”: {
“grpc”: 8502,
“https”: 8501
},
“retry_join”: [
“IP_OF_SERVER01”,
“IP_OF_SERVER02”,
“IP_OF_SERVER03”
],
“server”: true,
“ui”: true,
“verify_incoming”: true,
“verify_outgoing”: true,
“verify_server_hostname”: true
}
So in order to verify if the certificates are good or not, I performed my curl command:
curl -vvvk --cert euw1-consul-agent-ca.pem --key /root/consul-agent-ca-key.pem https://XXXXX:8501/v1/agent/connect/ca/roots --cacert euw1-consul-agent-ca.pem
And the certificate is verified as I have the message “SSL certificate verify ok”:
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=server.euw1.consul
* start date: Sep 1 06:26:40 2020 GMT
* expire date: Jun 17 07:26:40 2294 GMT
* issuer: C=US; ST=CA; L=San Francisco; street=101 Second Street; postalCode=94105; O=HashiCorp Inc.; CN=Consul Agent CA 251292510108551272556755631721680811671
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5649639eef50)
> GET /v1/agent/connect/ca/roots HTTP/2
> Host: XXXXX:8501
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< age: 0
< content-type: application/json
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-index: 3828
< x-consul-knownleader: false
< x-consul-lastcontact: 0
< content-length: 1435
< date: Tue, 01 Sep 2020 06:45:31 GMT
So without TLS Handshake issues, I don’t understand why connect-inject is not able to verify the servers.
I check the secrets for the tls.crt and tls.key and they are the right ones.
Following the same issue here Remote error: tls: bad certificate for K8S consul clients I tried to make the same with ACLs and the issue was the same “tls bad certificate”.
My helm values.yaml are:
global:
enabled: false
name: consul
domain: consul
image: "consul:1.8.3"
imagePullSecrets: []
imageK8S: "hashicorp/consul-k8s:0.18.1"
imageEnvoy: "envoyproxy/envoy-alpine:v1.14.2"
datacenter: euw1
enablePodSecurityPolicies: false
gossipEncryption:
secretName: "consul-secrets"
secretKey: "gossipEncryptionKey"
tls:
enabled: true
enableAutoEncrypt: true
serverAdditionalDNSSANs: []
serverAdditionalIPSANs: []
verify: true
httpsOnly: true
caCert:
secretName: "consul-secrets"
secretKey: tls.crt
caKey:
secretName: "consul-secrets"
secretKey: tls.key
enableConsulNamespaces: false
acls:
manageSystemACLs: true
bootstrapToken:
secretName: consul-secrets
secretKey: token_mgmt
createReplicationToken: false
replicationToken:
secretName: null
secretKey: null
federation:
enabled: false
createFederationSecret: false
lifecycleSidecarContainer:
resources:
requests:
memory: "25Mi"
cpu: "20m"
limits:
memory: "50Mi"
cpu: "20m"
server:
enabled: "-"
image: null
replicas: 3
enterpriseLicense:
secretName: null
secretKey: null
storage: 10Gi
storageClass: null
connect: true
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
updatePartition: 0
disruptionBudget:
enabled: true
maxUnavailable: null
extraConfig: |
{}
extraVolumes: []
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: server
topologyKey: kubernetes.io/hostname
tolerations: ""
nodeSelector: null
priorityClassName: ""
extraLabels: null
annotations: null
service:
annotations: null
extraEnvironmentVars: {}
disableFsGroupSecurityContext : false
externalServers:
enabled: true
hosts:
- [REDACTED]
httpsPort: 8501
tlsServerName: null
useSystemRoots: false
k8sAuthMethodHost: https://172.16.0.82
client:
enabled: true
image: null
join:
- [REDACTED]
dataDirectoryHostPath: null
grpc: true
exposeGossipPorts: true
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
extraConfig: |
{
"auto_encrypt": {
"tls": true
}
}
extraVolumes:
- type: secret
name: consul-secrets
tolerations: ""
nodeSelector: null
affinity: |
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: DoesNotExist
priorityClassName: "consul"
annotations: null
extraEnvironmentVars: {}
dnsPolicy: null
hostNetwork: false
updateStrategy: |
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
snapshotAgent:
enabled: false
replicas: 2
configSecret:
secretName: null
secretKey: null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
caCert: null
dns:
enabled: "-"
clusterIP: null
annotations: null
ui:
enabled: "-"
service:
enabled: true
type: null
annotations: null
additionalSpec: null
syncCatalog:
enabled: false
image: null
toConsul: true
toK8S: true
k8sPrefix: null
k8sAllowNamespaces: ["*"]
k8sDenyNamespaces: ["kube-system", "kube-public"]
k8sSourceNamespace: null
consulNamespaces:
consulDestinationNamespace: "default"
mirroringK8S: false
mirroringK8SPrefix: ""
addK8SNamespaceSuffix: true
consulPrefix: null
k8sTag: null
syncClusterIPServices: true
nodePortSyncType: ExternalFirst
aclSyncToken:
secretName: null
secretKey: null
nodeSelector: null
affinity: null
tolerations: null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
logLevel: info
consulWriteInterval: null
connectInject:
enabled: true
imageConsul: null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
imageEnvoy: null
namespaceSelector: null
k8sAllowNamespaces: ["*"]
k8sDenyNamespaces: []
consulNamespaces:
consulDestinationNamespace: "default"
mirroringK8S: false
mirroringK8SPrefix: ""
certs:
secretName: null
caBundle: ""
certName: tls.crt
keyName: tls.key
nodeSelector: null
affinity: null
tolerations: null
aclBindingRuleSelector: "serviceaccount.name!=default"
overrideAuthMethodName: ""
aclInjectToken:
secretName: null
secretKey: null
centralConfig:
enabled: true
defaultProtocol: null
proxyDefaults: |
{}
sidecarProxy:
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 100Mi
cpu: 100m
initContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
meshGateway:
enabled: false
globalMode: local
replicas: 2
wanAddress:
source: "Service"
port: 443
static: ""
service:
enabled: true
type: LoadBalancer
port: 443
nodePort: null
annotations: null
additionalSpec: null
imageEnvoy: envoyproxy/envoy-alpine:v1.14.2
hostNetwork: false
dnsPolicy: null
consulServiceName: "mesh-gateway"
containerPort: 8443
hostPort: null
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
initCopyConsulContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: mesh-gateway
topologyKey: kubernetes.io/hostname
tolerations: null
nodeSelector: null
priorityClassName: ""
annotations: null
ingressGateways:
enabled: false
defaults:
replicas: 2
service:
type: ClusterIP
ports:
- port: 8080
nodePort: null
- port: 8443
nodePort: null
annotations: null
additionalSpec: null
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
initCopyConsulContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: ingress-gateway
topologyKey: kubernetes.io/hostname
tolerations: null
nodeSelector: null
priorityClassName: ""
annotations: null
consulNamespace: "default"
gateways:
- name: ingress-gateway
terminatingGateways:
enabled: false
defaults:
replicas: 2
extraVolumes: []
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
initCopyConsulContainer:
resources:
requests:
memory: "25Mi"
cpu: "50m"
limits:
memory: "150Mi"
cpu: "50m"
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template "consul.name" . }}
release: "{{ .Release.Name }}"
component: terminating-gateway
topologyKey: kubernetes.io/hostname
tolerations: null
nodeSelector: null
priorityClassName: ""
annotations: null
consulNamespace: "default"
gateways:
- name: terminating-gateway
tests:
enabled: true
So are you guys were able to make this kind of setup correctly? @ashwin-venkatesh @ishustava @lkysow any help please?