Setting up ingress gateways using Nomad

Hi all,

I’ve been pulling my hair out trying to get this to work for the past week and would hugely appreciate any insights into what I’m doing wrong.

I’ve got a Consul + Nomad cluster that looks like the following:

Some notes about the network:

  • All Consul and Nomad servers and agents are on the same Tailscale network (not sure if that makes a difference)
  • Nomad does not have CA or encryption keys configured but the CNI plugin is installed on all Nomad clients.
  • Consul does not have ACLs set up, but CA and encryption keys are configured.
  • Envoy is not installed on any node

I wanted to create an ingress gateway per-datacenter, so I used this example Nomad service configuration file: nomad-connect-examples/ig-bridge-demo.nomad at master · hashicorp/nomad-connect-examples · GitHub with the following addition to ensure the service ran on Linux nodes:

  # Only run on Debian nodes
  constraint {
    attribute = "${}"
    value     = "linux"

It seemed to work the first time, correctly acting as a reverse proxy on uuid-api.ingress.dc1.consul.

However, I wanted the ingress gateway to act as a reverse proxy for all services hosted in the datacenter, and reading the documentation
led me to believe that I could proxy all services by using name = * and protocol = http instead of tcp.

I updated my ingress configuration to look as follows:

          # Consul Ingress Gateway Configuration Entry.
          ingress {
            # Nomad will automatically manage the Configuration Entry in Consul
            # given the parameters in the ingress block.
            # Additional options are documented at
            listener {
              port     = 8080
              protocol = "http"
              service {
                name = "uuid-api"

And then tried to redeploy but I got this error:

Error writing config entry ingress-gateway/ingress-ngproxy: Unexpected response code: 500 (rpc error making call: service "count-dashboard" has protocol "tcp", which does not match defined listener protocol "http")

This led to me trying to write a new config for proxy-defaults and writing it with consul config write:

  "Kind": "proxy-defaults",
  "Name": "global",
  "Config": {
      "Protocol": "http"

After doing this and re-deploying services, the ingress gateway ceased to work. Reverting the configuration didn’t help either, nor did restarting Consul/Nomad or the server/clients. When adding the job, the following line sticks out:

Feb 13 23:24:45 gamma-compute consul[48760]: 2023-02-13T23:24:45.208Z [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=_nomad-task-08d70a6e-3a53-6b54-c267-bcf828a32ad8-group-ingress-group-my-ingress-service-8080 service_id=_nomad-task-08d70a6e-3a53-6b54-c267-bcf828a32ad8-group-ingress-group-my-ingress-service-8080 id=gateway-config error="invalid type for config entry: <nil>"

I’ve since also tried writing service-defaults:

  "Kind": "service-defaults",
  "Name": "uuid-api",
  "Protocol": "http"

But Nomad just seems to straight up ignore the config, with the following error:

Error submitting job: Unexpected response code: 500 (rpc error: rpc error: Unexpected response code: 500 (service "uuid-api" has protocol "tcp", which does not match defined listener protocol "http"))

If I deploy the uuid-api service separately, it doesn’t seem to work at all, even if I try to access it on the host-allocated port instead of the reverse proxy.

I’m feeling a bit lost, and not sure where to go or what else to try. Any help would be appreciated :grinning:

Possibly related GitHub issues and PRs: