Raspberry Pi 4 8GB rev 1.4
Ubuntu 20.04
Nomad v1.0.4 (9294f35f9aa8dbb4acb6e85fa88e3e2534a3e41a)
Consul v1.9.5 Revision 3c1c22679
installed by ansible role community with consul connect enabled.
When I try a nomad job without consul connect instructions, no problem, it works.
But, I want to learn about consul connect, I follow the guide Consul Connect | Nomad by HashiCorp for CNI plugin. I download the arm64 version.
But when I try to apply consul connect service in nomad job and launch one, I have this error on each connect-proxy-* task.
standard_init_linux.go:219: exec user process caused: exec format error
standard_init_linux.go:219: exec user process caused: exec format error
standard_init_linux.go:219: exec user process caused: exec format error
[...]
Consul Connect uses a sidecar proxy that is automatically deployed by Nomad alongside your jobs, that’s the connect-proxy-* tasks that you see failing. This sidecar is a proxy called Envoy that Consul automatically configures to enable the Connect magic sauce.
By default, Nomad will use an official Envoy Docker image that is compatible with the Consul version that you are running. Since you are running Consul 1.9.5, the Envoy version that Nomad would use is 1.16.2, which does have an arm64 image.
Could you check which version of the Envoy image is being used?
You can use the nomad job inspect command and look for the connect-proxy-* tasks to see which images they are using.
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/linuxserver/piwigo latest 83fe89c4831b 36 hours ago 301MB
ghcr.io/linuxserver/mariadb latest 4fc227ba1b8d 11 days ago 343MB
traefik latest db4a19369dcb 2 weeks ago 85.5MB
envoyproxy/envoy v1.11.2 72e91d8680d8 19 months ago 150MB
gcr.io/google_containers/pause-arm64 3.1 6cf7c80fe444 3 years ago 525kB
envoy 1.11.2 ?!
It’s fresh install of nomad / consul, I didn’t pull envoy before.
Do I need to force pull new version?
or is it a problem with arm64 deploy?
1.11.2 is a bit old, and not really supported by Consul, so it’s very strange that it got picked
The 1.11.2 legacy envoy version is the fallback option when the version of Consul is too old to support that /v1/agent/selfxDS response object. Is it possible the job was initially launched using an older version of Consul?
For reference, envoy didn’t have arm64 images until v1.16
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9540 0 9540 0 0 1552k 0 --:--:-- --:--:-- --:--:-- 1552k
null
So, I added to client.hcl
meta {
"connect.sidecar_image" = "envoyproxy/envoy:v1.16.2"
}
It works! But…
This is my two services configs in nomad job:
service {
name = "piwigo-dev-lan"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "piwigo-db-dev-lan"
local_bind_port = 3306
}
}
}
}
port = "webinterface"
tags = [
"traefik.enable=true",
"traefik.http.routers.piwigo.entrypoints=http",
"traefik.http.routers.piwigo.rule=Host(`piwigo.dev.lan`)",
"traefik.http.routers.piwigo.service=piwigo-dev-lan",
]
}
service {
name = "piwigo-db-dev-lan"
port = "database"
tags = []
connect {
sidecar_service {}
}
}
In web install, when I put localhost or localhost:3306 , unable to connect.
In connect-proxy-db… stderr I have this error in loop:
[2021-05-06 06:03:54.488][10][warning][config] [bazel-out/aarch64-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2021-05-06 06:03:55.542][10][warning][config] [bazel-out/aarch64-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2021-05-06 06:04:07.908][10][warning][config] [bazel-out/aarch64-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
stderrhave lot of informations, but I don’t know If I can paste it here.
Nice! You can remove that meta attribute from you Nomad client now if you want. Nomad should retrieve the right image now.
From your service name, I’m assuming you are trying to run this Piwigo project. I don’t know much about it, but I was able to run the web install using this job:
Please note that this was just a test and you will most likely need to work a bit more on it
To connect to the database I had to use 127.0.0.1 instead of localhost. It seems like the PHP client for MySQL treats these two values as different things:
Note :
Whenever you specify “localhost” or “localhost:port” as server, the MySQL client library will override this and try to connect to a local socket (named pipe on Windows). If you want to use TCP/IP, use “127.0.0.1” instead of “localhost”. If the MySQL client library tries to connect to the wrong local socket, you should set the correct path as in your PHP configuration and leave the server field blank.
Just make sure you are properly persisting the MySQL data (check these guides on stateful workloads) and be careful with storing passwords in your job file.