Setting up ingress gateways using Nomad

Nomad
,
1

Hi all,

I’ve been pulling my hair out trying to get this to work for the past week and would hugely appreciate any insights into what I’m doing wrong.

I’ve got a Consul + Nomad cluster that looks like the following:

image
image1920×502 26.7 KB

Some notes about the network:

  • All Consul and Nomad servers and agents are on the same Tailscale network (not sure if that makes a difference)
  • Nomad does not have CA or encryption keys configured but the CNI plugin is installed on all Nomad clients.
  • Consul does not have ACLs set up, but CA and encryption keys are configured.
  • Envoy is not installed on any node

I wanted to create an ingress gateway per-datacenter, so I used this example Nomad service configuration file: nomad-connect-examples/ig-bridge-demo.nomad at master · hashicorp/nomad-connect-examples · GitHub with the following addition to ensure the service ran on Linux nodes:

  # Only run on Debian nodes
  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

It seemed to work the first time, correctly acting as a reverse proxy on uuid-api.ingress.dc1.consul.

However, I wanted the ingress gateway to act as a reverse proxy for all services hosted in the datacenter, and reading the documentation
led me to believe that I could proxy all services by using name = * and protocol = http instead of tcp.

I updated my ingress configuration to look as follows:

          # Consul Ingress Gateway Configuration Entry.
          ingress {
            # Nomad will automatically manage the Configuration Entry in Consul
            # given the parameters in the ingress block.
            #
            # Additional options are documented at
            # https://www.nomadproject.io/docs/job-specification/gateway#ingress-parameters
            listener {
              port     = 8080
              protocol = "http"
              service {
                name = "uuid-api"
              }
            }
          }

And then tried to redeploy but I got this error:

Error writing config entry ingress-gateway/ingress-ngproxy: Unexpected response code: 500 (rpc error making call: service "count-dashboard" has protocol "tcp", which does not match defined listener protocol "http")

This led to me trying to write a new config for proxy-defaults and writing it with consul config write:

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "Config": {
      "Protocol": "http"
  }
}

After doing this and re-deploying services, the ingress gateway ceased to work. Reverting the configuration didn’t help either, nor did restarting Consul/Nomad or the server/clients. When adding the job, the following line sticks out:

Feb 13 23:24:45 gamma-compute consul[48760]: 2023-02-13T23:24:45.208Z [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=_nomad-task-08d70a6e-3a53-6b54-c267-bcf828a32ad8-group-ingress-group-my-ingress-service-8080 service_id=_nomad-task-08d70a6e-3a53-6b54-c267-bcf828a32ad8-group-ingress-group-my-ingress-service-8080 id=gateway-config error="invalid type for config entry: <nil>"

I’ve since also tried writing service-defaults:

{
  "Kind": "service-defaults",
  "Name": "uuid-api",
  "Protocol": "http"
}

But Nomad just seems to straight up ignore the config, with the following error:

Error submitting job: Unexpected response code: 500 (rpc error: rpc error: Unexpected response code: 500 (service "uuid-api" has protocol "tcp", which does not match defined listener protocol "http"))

If I deploy the uuid-api service separately, it doesn’t seem to work at all, even if I try to access it on the host-allocated port instead of the reverse proxy.

I’m feeling a bit lost, and not sure where to go or what else to try. Any help would be appreciated :grinning:

Possibly related GitHub issues and PRs: