Error: "dial tcp: address 2601:547:980:b3e0::677c:32774: too many colons in address"

Consul
1

I got this “dial tcp: address 2601:547:980:b3e0::677c:32774: too many colons in address” error today and would like to see if anyone could help me. More details follow.

My Environment

  • Ubuntu 18.04

  • Consul 1.8.4 and 1.6.3 [1]

  • consul agent arguments:

    • -raft-protocol=3
    • -config-dir=/etc/ywen/consul/consul.d
    • -data-dir=/var/lib/consul/data.d
    • -enable-local-script-checks
    • -server
    • -node-id=232377e0-de74-4088-9647-e0d04f6cdf90
    • -ui
    • -bind 2601:547:980:b3e0::677c
    • -client ::1 127.0.0.1 172.28.128.1 192.0.2.1 198.51.100.1

  • We use Registrator to register Docker containers as services to Consul.

  • We use Ansible to configure the target machine.

The error was about TCP-checking the health of a postgres:13 Docker container which was set up in our Ansible playbook this way:

docker_container:
    env:
      SERVICE_CHECK_TCP: "true"
      SERVICE_CHECK_INTERVAL: 15s
      SERVICE_CHECK_TIMEOUT: 1s
   # Other `docker_container` arguments.

My Efforts

  • I found the PR " Ensure Consul is IPv6 compliant" but this bug was fixed as early as 1.5.2. I’m assuming what I’m encountering is not a regressional issue.
  • I confirmed that the PostgreSQL container was running properly when this error happened.
  • I ran pg_isready to test the database connection and it reported “good”:
ywen@ywen-Precision-7510:/tmp$ pg_isready --host=2601:547:980:b3e0::677c --port=32774
2601:547:980:b3e0::677c:32774 - accepting connections
  • I’m guessing that when Registrator used the IPv6 address to register the container to Consul and, perhaps, somehow Consul didn’t handle the registration properly(??) so the IPv6 address was not quoted correctly. I’m guessing so because I tried to change the -bind argument to my IPv4 address but didn’t fix the error. Consul still reported health check failure with the IPv6 address as shown above.

Could someone help me? Could this be some misconfiguration somewhere I’m not aware of?

Notes

  • [1] My project uses 1.6.3 which was released in Jan 2020. I tried the latest 1.8.4 today after I found this error and still got the same error.
2

Hi @yaobinwen-mvs,

The IPv6 address used in the check should utilize IPv6 bracketed notation to allow Consul to distinguish between the address, and port (e.g., [2601:547:980:b3e0::677c]:32774).

Looking at the code for Registrator (gliderlabs/registrator/consul/consul.go#119-120), I can see the value of the check’s TCP field is just the service’s IP and port separated by a colon (i.e., 2601:547:980:b3e0::677c:32774), which is incorrect.

According to the docs for Registrator’s Service Object under the section IP and port, its possible to override the IP address used in service registrations via the -ip flag.

Try setting this flag to -ip="[2601:547:980:b3e0::677c]" when you run Registrator, and see if that resolves your issue.

For example:

$ docker run -d \
    --name=registrator \
    --net=host \
    --volume=/var/run/docker.sock:/tmp/docker.sock \
    gliderlabs/registrator:latest \
    -ip="[2601:547:980:b3e0::677c]" \
    consul://localhost:8500
3

Hi @blake Thanks for the reply!

Your suggestion of using -ip with a properly wrapped IPv6 address works. And for anyone that uses Ansible to handle IP address, Ansible has the ipwrap filter that handles the wrapping elegantly.

Thanks for the link to Registrator’s source code. Do you think it is a bug in Registrator that they should have wrapped IPv6 addresses properly?

4

Hi @yaobinwen-mvs,

Glad to hear this solution worked! I think this is something that Registrator could easily fix by using net.JoinHostPort instead of the current IP and port formatting function.

1 Like
5

FYI: I just reported this in Registrator's issue list: https://github.com/gliderlabs/registrator/issues/691

Will watch how it goes.