Nomad v0.12.9 and consul 1.9.0 service mesh = Envoy 1.11.2 is too old and is not supported by Consul

v01t · December 2, 2020, 1:49am

Hello,

after successful test of traefik load balancing configuration (https://learn.hashicorp.com/tutorials/nomad/load-balancing-traefik), when traefik retrieve instance information using consul provider,

I decided to try with service mesh (https://learn.hashicorp.com/tutorials/nomad/consul-service-mesh#run-a-connect-enabled-job) configuration, following the tutorial I discover that I can’t reach services that registered in consul by nomad because some consul health-check’s was failing:

also I do not see any port listeners for instances (ip:port) that are registered in consul, future investigation lead me to envoyproxy container log with following information:

[warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 3, Envoy 1.11.2 is too old and is not supported by Consul

not sure if that’s the root cause of my problem, but based on used image version, it seems to be triggered here:

github.com

hashicorp/nomad/blob/b3c8deeac443696c8e240464829338314f4e4baa/client/allocrunner/taskrunner/envoy_version_hook.go#L147


	image := h.taskImage(config)
	return strings.Contains(image, structs.EnvoyVersionVar)
}
// image determines the best Envoy version to use. If supported is nil or empty
// Nomad will fallback to the legacy envoy image used before Nomad v1.0.
func (_ *envoyVersionHook) tweakImage(configured string, supported map[string][]string) (string, error) {
	versions := supported["envoy"]
	if len(versions) == 0 {
		return envoyLegacyImage, nil
	}
	latest, err := semver(versions[0])
	if err != nil {
		return "", err
	}
	return strings.ReplaceAll(configured, structs.EnvoyVersionVar, latest), nil
}

future investigation shows that used envoy version are not supported by latest (1.9) consul:

can this “warning” be a reason for failing connection?
can I overtake this situation somehow or I need to downgrade to consul 1.8.6 for now?

Thanks.

mcfarlandj · December 2, 2020, 3:39pm

Yes, we ran into the same issue. You can add a connect.sidecar_image metadata value to force a specific version. In Nomad 1.0.0 they have used the supported proxies API in Consul 1.8.6 and later to update this dynamically, but in earlier versions we have to do it manually.

    client {
      enabled = true
      meta = {
        connect.sidecar_image = "envoyproxy/envoy:v1.14.5"
      }
    }

shoenig · December 2, 2020, 6:18pm

Indeed, the upcoming Nomad v1.0.0 release will significantly improve the compatibility story between Nomad, Consul, and Envoy by automatically choosing the latest version of Envoy supported by Consul at runtime. You can read more about the change in the 1.0 upgrade notes.

In the interim you could do one of a few things

Set the sidecar image manually as suggested by @mcfarlandj
Downgrade Consul to a version prior to 1.9
Upgrade to Nomad 1.0.0-beta3

v01t · December 2, 2020, 7:10pm

Hello and many thanks @mcfarlandj and @shoenig,

setting sidecar envoy version resolve consul health-check / listeners,
and I was able to connect dashboard (ui) and api (backend) from countdash example, one issue with that, so far, is that sometimes when you reload the page UI disconnects from back-end and it may take up to 5-7 seconds to reconnect back (not sure if that’s just example specifics or some problems in my configuration, will try with some more examples and/or nomad update to clarify that.

Thanks again for such useful replies.