In some efforts related to circumventing Request for Debugging Recommendation: NXDomain on Nomad w/ Consul Connect deployment DNS query, I’ve discovered/created an issue that causes my Consul template interpolation to fail.
My job spec remains roughly as in the previous post. I’m using consul-template to write out an Nginx reverse config proxy that has worked before.
The Consul template for an Nginx configuration is constructed as follows:
%{ for service, port in exposed_services ~}
cat >> $${LOAD_BALANCER_TEMPLATE} << EOF
upstream ${service} {
least_conn;
{{- range service "${service}" }} # range connect produces the same
server {{ .Address }}:{{ .Port }};
{{- end }}
}
server {
listen ${port};
location / {
proxy_pass http://${service};
}
}
EOF
%{ endfor ~}
A simple deployment of one service, myservice, renders as:
upstream myservice {
least_conn;
# no servers found here : (
}
server {
listen 80;
location / {
proxy_pass http://myservice;
}
}
So, clearly the service ranges are not populating. However, there are a few other underlying issues in the actual deployment which concern me quite a bit and I believe to be the real problem.
Consul DNS does not recognize the service
As noted in the previous post, I cannot query for the service against the Consul DNS. I’ve tried a good many DNS configuration options at this point, so I really don’t know what I changed.
Consul’s registration of the service seems odd
First of all, all of my server agents only recognize the Nomad service. I had assumed in the past that this was just do to how Consul does its book keeping. But, at this point, I’m thinking I don’t know much at all.
# curl http://127.0.0.1:8500/v1/agent/services
{
"_nomad-server-b5eta6ejsrmloarykwqlbyezb5drd2bv": {
"ID": "_nomad-server-b5eta6ejsrmloarykwqlbyezb5drd2bv",
"Service": "nomad",
"Tags": [
"http"
],
"Meta": {
"external-source": "nomad"
},
"Port": 4646,
"Address": "10.128.0.3",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.128.0.3",
"Port": 4646
},
"wan_ipv4": {
"Address": "10.128.0.3",
"Port": 4646
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Datacenter": "us-west1"
},
"_nomad-server-bazmqhtkfnaw7mmtqsihuj2curhlbd6s": {
"ID": "_nomad-server-bazmqhtkfnaw7mmtqsihuj2curhlbd6s",
"Service": "nomad",
"Tags": [
"rpc"
],
"Meta": {
"external-source": "nomad"
},
"Port": 4647,
"Address": "10.128.0.3",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.128.0.3",
"Port": 4647
},
"wan_ipv4": {
"Address": "10.128.0.3",
"Port": 4647
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Datacenter": "us-west1"
},
"_nomad-server-ggwmr3qndnw43jhojh42ast7wrldk75h": {
"ID": "_nomad-server-ggwmr3qndnw43jhojh42ast7wrldk75h",
"Service": "nomad",
"Tags": [
"serf"
],
"Meta": {
"external-source": "nomad"
},
"Port": 4648,
"Address": "10.128.0.3",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.128.0.3",
"Port": 4648
},
"wan_ipv4": {
"Address": "10.128.0.3",
"Port": 4648
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Datacenter": "us-west1"
}
}
I can get the service configuration by addressing it directly:
[
{
"ID": "ea832024-ea2f-2d7e-b2cb-305c1db0450b",
"Node": "gcp-rpc-cluster-clients-g2nc",
"Address": "10.128.0.6",
"Datacenter": "us-west1",
"TaggedAddresses": {
"lan": "10.128.0.6",
"lan_ipv4": "10.128.0.6",
"wan": "10.128.0.6",
"wan_ipv4": "10.128.0.6"
},
"NodeMeta": {
"consul-network-segment": ""
},
"ServiceKind": "ingress-gateway",
"ServiceID": "_nomad-task-9f707188-49a6-064d-0ae7-e70270b73e11-group-rpc-myservice-http",
"ServiceName": "myservice",
"ServiceTags": [],
"ServiceAddress": "10.128.0.6",
"ServiceTaggedAddresses": {
"lan_ipv4": {
"Address": "10.128.0.6",
"Port": 24888
},
"wan_ipv4": {
"Address": "10.128.0.6",
"Port": 24888
}
},
"ServiceWeights": {
"Passing": 1,
"Warning": 1
},
"ServiceMeta": {
"external-source": "nomad"
},
"ServicePort": 24888,
"ServiceSocketPath": "",
"ServiceEnableTagOverride": false,
"ServiceProxy": {
"Mode": "",
"Config": {
"connect_timeout_ms": 5000,
"envoy_gateway_bind_addresses": {
"myservice": {
"Address": "0.0.0.0",
"Port": 889
}
},
"envoy_gateway_no_default_bind": true
},
"MeshGateway": {},
"Expose": {}
},
"ServiceConnect": {},
"CreateIndex": 166,
"ModifyIndex": 166
}
]
Interestingly the proxy port does not serve the service. I can only use the service if I interact directly with the dynamically allocated port. I can confirm there is no firewall in the way.
What gives?
So, the situation appears to be thus:
- I made some changes which I cannot reverse.
- Consul DNS does not recognize my services.
- Changes to DNS configurations on the host do not seem to make a difference.
- Proxies do not provide their associated services.
- The services are not available to consul-template.
What can I try next?