I am working on implementing Consul with end-to-end TLS in our environment, and have run into an issue similar to the one that @thecosmicfrog experienced in this thread. Kubernetes init containers used in the Connect pod and CatalogSync pod are failing to get the CA root certificates from the Consul CA. However, the workaround presented by [user] doesn’t seem to be viable for me.
The Consul architecture is 3 servers installed on Ubuntu 18.04 VMs, with Consul configured as client-only on our Kubernetes cluster. I can get Consul to securely discover services on the cluster, so long as they have the Consul annotations in their deployment spec. However, when trying to enable Consul Connect or CatalogSync, the injector pods get stuck in Init:0/1. The container getting stuck is get-auto-encrypt-client-ca, with error 2020-07-21T16:16:51.166Z [ERROR] Error retrieving CA roots from Consul: err="Get "https://<ip>:8501/v1/agent/connect/ca/roots": dial tcp <ip>:8501: connect: connection refused"
Reading the above thread, I tried execing into the containers to get the certificates copied into the container CA store, but the containers are running as nonroot, so I can’t get the files copied from/consul/tls/ca/ to /usr/local/share/ca-certificates/. It looks like I could possibly adjust the Helm _helpers.tpl file to do it as part of the container’s startup script, but I’m not sure if that’s a good idea.
Helm Chart (disabled parts redacted where indicated)…I have tried enabling certs.secretName and using the consul-agent-ca.pem as the caBundle, but it makes no difference.
global:
enabled: true
name: consul
domain: consul
image: "consul:1.8.0"
imagePullSecrets: []
imageK8S: "hashicorp/consul-k8s:0.17.0"
imageEnvoy: envoyproxy/envoy:v1.14.2
datacenter: sv-consul
enablePodSecurityPolicies: false
gossipEncryption:
secretName: "consul-gossip-encryption-key"
secretKey: "key"
tls:
enabled: true
enableAutoEncrypt: true
serverAdditionalDNSSANs: []
serverAdditionalIPSANs: []
verify: false
httpsOnly: false
caCert:
secretName: consul-ca-cert
secretKey: tls.crt
caKey:
secretName: consul-ca-key
secretKey: tls.key
enableConsulNamespaces: false
acls:
manageSystemACLs: false
bootstrapToken:
secretName: null
secretKey: null
createReplicationToken: false
replicationToken:
secretName: null
secretKey: null
federation:
enabled: false
createFederationSecret: false
server:
enabled: false
<redacted>
externalServers:
enabled: true
hosts: [<redacted>]
httpsPort: 8501
tlsServerName: null
useSystemRoots: false
k8sAuthMethodHost: null
client:
enabled: true
image: "consul:1.8.0"
join: [<redacted>]
dataDirectoryHostPath: null
grpc: true
exposeGossipPorts: true
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"
extraConfig: |
{}
extraVolumes: []
tolerations: ""
nodeSelector: null
affinity: {}
priorityClassName: ""
annotations: null
extraEnvironmentVars: {}
dnsPolicy: null
hostNetwork: false
updateStrategy: null
snapshotAgent:
enabled: false
replicas: 2
configSecret:
secretName: null
secretKey: null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
caCert: null
dns:
enabled: "-"
clusterIP: null
annotations: null
ui:
enabled: "-"
service:
enabled: true
type: null
annotations: null
additionalSpec: null
syncCatalog:
enabled: true
image: null
default: true # true will sync by default, otherwise requires annotation
toConsul: true
toK8S: false
k8sPrefix: null
k8sAllowNamespaces: ["*"]
k8sDenyNamespaces: ["kube-system", "kube-public"]
k8sSourceNamespace: null
consulNamespaces:
consulDestinationNamespace: "default"
mirroringK8S: false
mirroringK8SPrefix: ""
addK8SNamespaceSuffix: true
consulPrefix: null
k8sTag: null
syncClusterIPServices: true
nodePortSyncType: ExternalFirst
aclSyncToken:
secretName: null
secretKey: null
nodeSelector: null
affinity: null
tolerations: null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
logLevel: info
consulWriteInterval: null
connectInject:
enabled: true
image: hashicorp/consul-k8s:latest # image for consul-k8s that contains the injector
default: false # true will inject by default, otherwise requires annotation
imageConsul: consul:1.8.0
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
imageEnvoy: envoyproxy/envoy:v1.14.2
namespaceSelector: null
k8sAllowNamespaces: ["*"]
k8sDenyNamespaces: []
consulNamespaces:
consulDestinationNamespace: "default"
mirroringK8S: false
mirroringK8SPrefix: ""
certs:
#secretName: consul-connect-inject-ca
secretName: null
#caBundle: <redacted>
caBundle: ""
certName: tls.crt
keyName: tls.key
nodeSelector: null
affinity: null
tolerations: null
aclBindingRuleSelector: "serviceaccount.name!=default"
overrideAuthMethodName: ""
aclInjectToken:
secretName: null
secretKey: null
centralConfig:
enabled: true
defaultProtocol: null
proxyDefaults: |
{}
sidecarProxy:
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 100Mi
cpu: 100m
meshGateway:
enabled: false
<redacted>
ingressGateways:
enabled: false
<redacted>
terminatingGateways:
enabled: false
<redacted>
tests:
enabled: true
consul.hcl on the servers. have tried flipping verify_* to false, but no effect. At some point we need verification anyway.
server = true
datacenter = "sv-consul"
data_dir = "/opt/consul"
encrypt = “<redacted>"
server = true
client_addr = "0.0.0.0"
ui = true
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/sv-consul-server-consul-0.pem"
key_file = "/etc/consul.d/sv-consul-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
auto_encrypt = {
allow_tls = true
}
retry_join = [<redacted>]
acl = {
enabled = true
default_policy = "allow"
enable_token_persistence = true
}
performance {
raft_multiplier = 1
}
server.hcl on the servers:
server = true
bootstrap_expect = 3
ui = true
client_addr = "0.0.0.0"
connect {
enabled = true
}
Any guidance would be appreciated.
Thanks,
Chris
