Unable to connect to service via host IP and port. Anything I'm missing here?

Nomad
1

I’m having difficulty connecting to a postgres service via $HOST_IP:5432 (error returned is Connection refused) with this network config

network {
  mode = "bridge"

  port "postgres" {
    static = 5432
    to = 5432
  }
}

Within the container, running with netstat -tlpn4 returns

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -

Is there anything else I’m missing here? Thanks

2

Hi @axsuul, can you describe the network topology a little more? Where are you trying to connect from? What does the Nomad Client network config look like? What does the alloc status report for its network configuration?

E.g. you should see something like,

# nomad alloc status <allocid>

Allocation Addresses
Label  Dynamic  Address
*db    yes      192.168.88.226:5432 -> 5432

And I’m able to connect to it from another host

➜ docker run -it --rm --net=host postgres:14 psql -h 192.168.88.226 -U postgres 
Password for user postgres: 
psql (14.4 (Debian 14.4-1.pgdg110+1))
Type "help" for help.

postgres=#

Using this toy postgres example

job "postgres-example" {
  datacenters = ["dc1"]

  group "database" {
    network {
      port "db" {
        static = 5432
        to     = 5432
      }
    }

    task "postgres" {
      driver = "docker"
      config {
        image = "postgres:14"
        ports = ["db"]
      }
      env {
        POSTGRES_PASSWORD = "abc123"
      }
      resources {
        cpu    = 100
        memory = 128
      }
    }
  }
}
3

Thanks for getting back to me @seth.hoenig

I am trying to connect using TablePlus, a postgres GUI client, from my workstation → server. TablePlus also offers a way to SSH in and connect locally but that also doesn’t work. I should mention that I have the same configuration deployed on a staging server and I can connect just fine there but not here on production — something seems to be different.

Here is my full Nomad client config but not seeing anything network related

datacenter = "main1"
data_dir = "/opt/nomad"

vault {
  enabled = true
  address = "http://vault.service.consul:8200"
}

consul {
  address = "127.0.0.1:8500"
}

# Tasks can also run on managers
client {
  enabled = true
  node_class = "member"

  template {
    # Allows templates to use files on host as source
    disable_file_sandbox = true
  }

  meta {
    node_groups = "postgres"
  }
}

plugin "docker" {
  config {
    # Allows containers to run in privileged mode
    allow_privileged = true

    auth {
      # Required to properly pull Docker images from Google Cloud Container Registry. Ensure that a `gcloud auth login`
      # has been performed as well.
      config = "/opt/nomad/docker.json"
    }

    volumes {
      # Allows tasks to bind host paths within container
      enabled = true
    }
  }
}

plugin "raw_exec" {
  config {
    # Needs to be enabled in order for us to use the raw_exec task driver
    enabled = true
  }
}

I forgot to mention I’m also using Consul Connect, if that matters. Here is the alloc status report

# nomad alloc status
Allocation Addresses (mode = "bridge")
Label                    Dynamic  Address
*postgres                yes      10.128.0.66:5432 -> 5432
*connect-proxy-postgres  yes      10.128.0.66:28984 -> 28984

Based on this I’ve also now tried

docker run -it --rm --net=host postgres:11 psql -h 10.128.0.66 -U postgres

which returns

psql: error: connection to server at "10.128.0.66", port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?

So far does anything look off?

4

Just wanted to follow up on my last post but understand you’re busy, thanks