I’m trying to connect Elastic Search in kubernetes with Consul. If I install everything without Consul, it all works as expected. If I try to install with the Consul service mesh enabled, I am unable to connect to the elastic search server from Filebeat. I can connect from Kibana following the setup mentioned here: Consul Connect and ECK · Issue #2973 · elastic/cloud-on-k8s · GitHub under " Associating through the mesh".
All services are installed using helm templates
Interestingly Kibana seems to connect OK to Elastic, but as FileBeat is outside of the mesh, there seems to be no way to get it to connect. There also seems to be problems checking the license.
I’m seeing the following error in the elastic search deployment:
Get "http://elastic-search-es-internal-http.elastic-system.svc:9200/_license": EOF
And the following error from FileBeat (whatever url I’ve tried gets the same error):
Failed to connect to backoff(elasticsearch(http://elastic-search-es-http.elastic-system.svc:9200)): Get "http://elastic-search-es-http.elastic-system.svc:9200": EOF
I’ve also seen that if I try to setup Elastic with transparent proxy in the same way as I have for Kibana, but it won’t deploy at all (I can’t see any useful error messages for this)
consul.hashicorp.com/transparent-proxy: "true"
consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: 9200
I’m sure I have missed something simple here, but if anyone has any thoughts, please let me know. I’ve spent almost a week here trying every option I can think of and got nowhere.
My configuration for Consul is as follows:
global:
name: consul
datacenter: dc1
metrics:
enabled: true
tls:
enabled: true
enableAutoEncrypt: true
verify: true
serverAdditionalDNSSANs:
## Add the K8s domain name to the consul server certificate
- "consul-server.consul-system.svc.cluster.local"
## For production turn on ACLs and gossipEncryption:
# acls:
# manageSystemACLs: true
# gossipEncryption:
# secretName: "consul-gossip-encryption-key"
# secretKey: "key"
server:
replicas: 1
securityContext:
runAsNonRoot: false
runAsUser: 0
ui:
enabled: true
connectInject:
# This method will inject the sidecar container into Pods:
enabled: true
# But not by default, only do this for Pods that have the explicit annotation:
# consul.hashicorp.com/connect-inject: "true"
default: false
controller:
enabled: true
prometheus:
enabled: true
grafana:
enabled: true
syncCatalog:
# This method will automatically synchronize Kubernetes services to Consul:
# (No sidecar is injected by this method):
enabled: true
# But not by default, only for Services that have the explicit annotation:
# consul.hashicorp.com/service-sync: "true"
default: false
# Synchronize from Kubernetes to Consul:
toConsul: true
# But not from Consul to K8s:
toK8S: true
Elastic:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-search
spec:
version: {{ .Values.elastic.version }}
http:
tls:
selfSignedCertificate:
disabled: true
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
podTemplate:
metadata:
annotations:
consul.hashicorp.com/connect-service: "elastic-search"
consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-port: "http"
spec:
automountServiceAccountToken: true
serviceAccount: elastic-search
Kibana:
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
spec:
version: {{ .Values.elastic.version }}
count: 1
http:
tls:
selfSignedCertificate:
disabled: true
config:
elasticsearch.hosts:
- http://127.0.0.1:9200
elasticsearch.username: elastic
elasticsearch.ssl.verificationMode: none
podTemplate:
metadata:
annotations:
consul.hashicorp.com/connect-service: "kibana"
consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-port: "http"
consul.hashicorp.com/connect-service-upstreams: "elastic-search:9200"
consul.hashicorp.com/transparent-proxy: "true"
consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: 5601,80,443
spec:
automountServiceAccountToken: true
serviceAccount: kibana
containers:
- name: kibana
env:
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-search-es-elastic-user
key: elastic
Beats:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
spec:
type: filebeat
version: {{ .Values.elastic.version }}
elasticsearchRef:
name: elastic-search
config:
output.elasticsearch:
hosts: ["http://127.0.0.1:9200", "http://elastic-search-es-http:9200"]
username: "elastic"
password: "924y07bumdibu20y1JP7b4iI" # "${ELASTICSEARCH_PASSWORD}"
ssl.verificationMode: none
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
multiline.pattern: '^{'
multiline.negate: true
multiline.match: after
processors:
# flattens the array to a single string
- script:
when:
has_fields: ['error.stack_trace']
lang: javascript
id: my_filter
source: >
function process(event) {
event.Put("error.stack_trace", event.Get("error.stack_trace").join("\n"));
}
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
max_depth: 10
overwrite_keys: true
- add_kubernetes_metadata:
in_cluster: true
daemonSet:
podTemplate:
metadata:
annotations:
consul.hashicorp.com/connect-service: "filebeat"
consul.hashicorp.com/connect-service-upstreams: "elastic-search:9200"
spec:
automountServiceAccountToken: true
serviceAccount: filebeat
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
securityContext:
runAsUser: 0
containers:
- name: filebeat
env:
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-search-es-elastic-user
key: elastic
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
), is there any easy way for pods which don’t have a service (like beats) to access the API gateway or an ingress using something like the dns entries inside kubernetes. We will have at least 3 clusters all needing to connect to the same Elasticsearch instance. Ideally I don’t want to set up external DNS entries.