So I followed the Docs
- https://learn.hashicorp.com/consul/kubernetes/mesh-gateways
- https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes
And was able to set up wan federation between the clusters, but when I tried to get the VM to federate to these clusters I wasn’t able to, I think its because of the envoy proxy not starting properly
I used the official helm chart and used the following values file for the primary datacenter
global:
name: consul
image: consul:1.8.0
imageK8S: hashicorp/consul-k8s:0.16.0
datacenter: dc1
federation:
enabled: true
createFederationSecret: true
tls:
enabled: true
meshGateway:
enabled: true
connectInject:
enabled: true
my server config file -
cert_file = "/<location>/consul/config/vm-dc-server-consul-0.pem"
key_file = "/<location>/consul/config/vm-dc-server-consul-0-key.pem"
ca_file = "/<location>/consul/config/consul-agent-ca.pem"
primary_gateways = ["<IP of mesh service>:443"]
# Other server settings
server = true
datacenter = "vm"
data_dir = "/<location>/consul/data"
enable_central_service_config = true
primary_datacenter = "dc1"
connect {
enabled = true
enable_mesh_gateway_wan_federation = true
}
verify_incoming_rpc = true
verify_outgoing = true
verify_server_hostname = true
ports {
https = 8501
http = 8500
grpc = 8502
}
the logs from the VM are -
2020-07-09T07:25:58.304Z [ERROR] agent.server: failed to establish leadership: error="Failed to set the intermediate certificate with the CA provider: could not verify intermediate cert against root: x509: certificate has expired or is not yet valid: current time 2020-07-09T07:25:58Z is before 2020-07-09T07:27:05Z"
2020-07-09T07:25:58.304Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-09T07:25:58.304Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-09T07:25:58.304Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-09T07:25:58.304Z [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-09T07:25:58.449Z [WARN] agent: Check socket connection failed: check=service:vm-gateway error="dial tcp 10.154.0.17:7051: connect: connection refused"
2020-07-09T07:25:58.449Z [WARN] agent: Check is now critical: check=service:vm-gateway
2020-07-09T07:26:01.820Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Internal.ServiceDump
2020-07-09T07:26:03.343Z [ERROR] agent.server: failed to establish leadership: error="Failed to set the intermediate certificate with the CA provider: could not verify intermediate cert against root: x509: certificate has expired or is not yet valid: current time 2020-07-09T07:26:03Z is before 2020-07-09T07:27:10Z"
2020-07-09T07:26:03.343Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:03.343Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:03.343Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:03.343Z [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-09T07:26:08.370Z [ERROR] agent.server: failed to establish leadership: error="Failed to set the intermediate certificate with the CA provider: could not verify intermediate cert against root: x509: certificate has expired or is not yet valid: current time 2020-07-09T07:26:08Z is before 2020-07-09T07:27:15Z"
2020-07-09T07:26:08.370Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:08.370Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:08.370Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:08.370Z [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-09T07:26:08.450Z [WARN] agent: Check socket connection failed: check=service:vm-gateway error="dial tcp 10.154.0.17:7051: connect: connection refused"
2020-07-09T07:26:08.450Z [WARN] agent: Check is now critical: check=service:vm-gateway
2020-07-09T07:26:13.397Z [ERROR] agent.server: failed to establish leadership: error="Failed to set the intermediate certificate with the CA provider: could not verify intermediate cert against root: x509: certificate has expired or is not yet valid: current time 2020-07-09T07:26:13Z is before 2020-07-09T07:27:20Z"
2020-07-09T07:26:13.397Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:13.397Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:13.397Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:13.397Z [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-09T07:26:16.740Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-1.dc1 has failed, no acks received
2020-07-09T07:26:17.731Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "127.0.0.1:51388": tls: first record does not look like a TLS handshake
2020-07-09T07:26:18.428Z [ERROR] agent.server: failed to establish leadership: error="Failed to set the intermediate certificate with the CA provider: could not verify intermediate cert against root: x509: certificate has expired or is not yet valid: current time 2020-07-09T07:26:18Z is before 2020-07-09T07:27:25Z"
2020-07-09T07:26:18.428Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:18.428Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:18.428Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:18.428Z [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-09T07:26:18.450Z [WARN] agent: Check socket connection failed: check=service:vm-gateway error="dial tcp 10.154.0.17:7051: connect: connection refused"
2020-07-09T07:26:18.450Z [WARN] agent: Check is now critical: check=service:vm-gateway
2020-07-09T07:26:18.976Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Internal.ServiceDump
2020-07-09T07:26:19.395Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Internal.ServiceDump
2020-07-09T07:26:21.925Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "127.0.0.1:51392": tls: first record does not look like a TLS handshake
2020-07-09T07:26:22.052Z [WARN] agent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Internal.ServiceDump
2020-07-09T07:26:23.455Z [ERROR] agent.server: failed to establish leadership: error="Failed to set the intermediate certificate with the CA provider: could not verify intermediate cert against root: x509: certificate has expired or is not yet valid: current time 2020-07-09T07:26:23Z is before 2020-07-09T07:27:30Z"
2020-07-09T07:26:23.455Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:23.455Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:23.455Z [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-09T07:26:23.455Z [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
I have checked the ports they are all available using netstat and the firewall configuration allows all the consul ports
envoy logs -
==> Registered service: vm-gateway
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:255] initializing epoch 0 (hot restart version=disabled)
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:257] statically linked extensions:
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.lua, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.dubbo_proxy.protocols: dubbo
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.thrift_proxy.transports: auto, framed, header, unframed
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2020-07-09 07:28:25.174][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.dubbo_proxy.route_matchers: default
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.udp_listeners: raw_udp_listener
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.dubbo_proxy.serializers: dubbo.hessian2
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.health_checkers: envoy.health_checkers.redis
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.resolvers: envoy.ip
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] http_cache_factory: envoy.extensions.http.cache.simple
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-07-09 07:28:25.175][4737][info][main] [external/envoy/source/server/server.cc:259] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2020-07-09 07:28:25.196][4737][warning][misc] [external/envoy/source/common/protobuf/utility.cc:198] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-07-09 07:28:25.197][4737][info][main] [external/envoy/source/server/server.cc:340] admin address: 127.0.0.1:19005
[2020-07-09 07:28:25.198][4737][info][main] [external/envoy/source/server/server.cc:459] runtime: layers:
- name: static_layer
static_layer:
envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1: true
envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name: true
envoy.deprecated_features:envoy.api.v2.Cluster.tls_context: true
[2020-07-09 07:28:25.198][4737][info][config] [external/envoy/source/server/configuration_impl.cc:103] loading tracing configuration
[2020-07-09 07:28:25.198][4737][info][config] [external/envoy/source/server/configuration_impl.cc:69] loading 0 static secret(s)
[2020-07-09 07:28:25.198][4737][info][config] [external/envoy/source/server/configuration_impl.cc:75] loading 1 cluster(s)
[2020-07-09 07:28:25.206][4737][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:167] cm init: initializing cds
[2020-07-09 07:28:25.208][4737][info][config] [external/envoy/source/server/configuration_impl.cc:79] loading 0 listener(s)
[2020-07-09 07:28:25.208][4737][info][config] [external/envoy/source/server/configuration_impl.cc:129] loading stats sink configuration
[2020-07-09 07:28:25.209][4737][info][main] [external/envoy/source/server/server.cc:554] starting main dispatch loop
[2020-07-09 07:28:25.210][4737][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2020-07-09 07:28:25.514][4737][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2020-07-09 07:28:26.474][4737][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
I have used the command
consul connect envoy -mesh-gateway -register \
-service "secondary-primary" \
-address "<your private address>:<port>" \
-wan-address "<your externally accessible address>:<port>"\
-admin-bind 127.0.0.1:19005
with the above command only 19005 port is available the other ports are not available on netstat, probing into the admin url's /ready endpoint returns LIVE
consul server on k8’s log -
2020-07-09T07:31:59.081Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=vm
2020-07-09T07:32:01.050Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=vm
2020-07-09T07:32:03.018Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to <vm-external-ip>:8302: read tcp 10.32.1.8:38300->10.32.2.14:8443: read: connection reset by peer
2020-07-09T07:32:03.514Z [INFO] agent.server.memberlist.wan: memberlist: Suspect simba.vm has failed, no acks received
2020-07-09T07:32:03.515Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send compound ping and suspect message to <vm-external-ip>:8302: read tcp 10.32.1.8:46742->10.32.1.7:8443: read: connection reset by peer
firewall allows port 8302 tcp and udp for the VM.
on k8’s when I execute
kubectl exec consul-server-1 -- consul catalog services -datacenter vm
I get the following error
Error listing services: Unexpected response code: 500 (Remote DC has no server currently reachable)
command terminated with exit code 1
But I’m able to setup consul connect proxy to a service on kubernetes and connect to it.
Please can anyone let me know what mistakes I have made?