Incorrect service IP registered with consul?

Nomad
1

Hey all,

It seems that nomad registers the wrong IP to consul for a service, for some context I’ve setup two host_networks:

host_network "wan" {
  interface = "eth0"
}

host_network "lan" {
  interface = "eth1"
}

On the service I have host_network = "lan":

network {
  mode = "bridge"
  port "http" {
    host_network = "lan"
  }
}

The allocation is running on the correct "lan" address:

$ nomad alloc status <id>
...
Allocation Addresses (mode = "bridge")
Label  Dynamic  Address
*http  yes      10.0.0.2:20379
...

The service is also reachable on (lan) 10.0.0.2:20379, but consul reports the wrong .ServiceAddress, it returns the public ip of the server I’m on:

$ curl consul.service.consul:8500/v1/catalog/service/echo
[
  {
    "ID": "f36e091e-490b-328f-dd7b-c8cc728c867a",
    "Node": "main-001",
    "Address": "10.0.0.2",
    "Datacenter": "ams3",
    "TaggedAddresses": {
      "lan": "10.0.0.2",
      "lan_ipv4": "10.0.0.2",
      "wan": "10.0.0.2",
      "wan_ipv4": "10.0.0.2"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceKind": "",
    "ServiceID": "_nomad-task-2868b378-08f5-126f-4e90-68d8083788fb-group-echo-echo-http",
    "ServiceName": "echo",
    "ServiceTags": [],
    "ServiceAddress": "<public-ip>",
    "ServiceTaggedAddresses": {
      "lan_ipv4": {
        "Address": "<public-ip>",
        "Port": 20379
      },
      "wan_ipv4": {
        "Address": "<public-ip>",
        "Port": 20379
      }
    },
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-source": "nomad"
    },
    "ServicePort": 20379,
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 30,
    "ModifyIndex": 30
  }
]

Note also that .ServiceTaggedAddresses contains lan_ipv4 with a public ip, not the lan ip.

Changing the service.address_mode to driver or host, has no effect on the address registered to consul.

Nomad Version

Nomad v0.12.3 (2db8abd9620dd41cb7bfe399551ba0f7824b3f61)

Consul Version

Consul v1.8.3
Revision a9322b9c7
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
2 Likes
2

I have the exact same issue. Services bind to the correct interface but consul returns the global ip addresses instead of the link-local ones. I also don’t believe it’s a consul issue since nomad itself, vault and consul all register with link-local addresses. @yields did you by chance find a solution or workaround? I’m quite stuck on this.

3

I found a solution! Setting network_mode to "host" in the service stanza! E.g.:

service {
  name = "rabbitmq-mqtt"
  
  check {
    name = "alive"
    type = "tcp"
    port = "mqtt"
    interval = "10s"
    timeout = "2s"
  }

  address_mode = "host"
}
1 Like