Connecting to Postgres/Redis on database server from Nomad docker container "Error: Connection reset by peer"

Consul
,
1

I have Consul/Nomad cluster and I want my Nomad docker containers be able to connect to Postgresql/Redis database (which located on another server) via Consul.

I installed Consul on my database server and enabled enable_local_script_checks like this:

advertise_addr = "192.168.90.6"
advertise_addr_wan = "192.168.90.6"
node_name = "dev-client-postgres"
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
dns_config = {
  enable_truncate = true
  only_passing = true
}
data_dir = "/opt/consul"
datacenter = "dev"
disable_update_check = false
domain = "consul"
enable_local_script_checks = false
enable_script_checks = false
encrypt = "xyzxyzxyzxyzxyzxyzxyzxyz"
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
tls = {
  defaults = {
    verify_incoming = true
    verify_outgoing = true
    ca_file = "/etc/consul.d/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/dev-client-consul-0.pem"
    key_file = "/etc/consul.d/dev-client-consul-0-key.pem"
  }
  grpc {
    verify_incoming   = false
  }
  internal_rpc {
    verify_server_hostname = true
  }
}
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens = {
    agent = "xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz"
  }
}
connect = {
  enabled = true
}
performance = {
  leave_drain_time = "5s"
  raft_multiplier = 1
  rpc_hold_timeout = "7s"
}
ports = {
  grpc = 8502
  grpc_tls = 8503
}
raft_protocol = 3
retry_interval = "30s"
retry_join = ["192.168.80.3"]
retry_max = 0
server = false
translate_wan_addrs = false
enable_local_script_checks = true

and then I registered a service by postgresql.hcl and generate a new token with blow policy:

sudo vim postgres-policy.hcl

service "postgresql" {
policy = "write"
}

agent_prefix "" {
policy = "read"
}

node_prefix "" {
policy = "read"
}

   consul acl policy create -name "postgres-policy" -description "Policy for PostgreSQL service" -token=xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz -rules @postgres-policy.hcl
   consul acl token create -description "PostgreSQL service token" -token=xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz -policy-name "postgres-policy"

sudo vim postgresql.hcl

service {
  name = "postgresql"
  tags = [
    "database",
    "postgres"
  ]
  address = "192.168.90.6"
  port    = 5432
  token   = "xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz"
  connect = { sidecar_service = {} }
  check {
    id       = "postgres-tcp-check"
    name     = "PostgreSQL TCP Check"
    tcp      = "192.168.90.6:5432"
    interval = "10s"
    timeout  = "1s"
  }
#  check {
#    id       = "postgres-status-check"
#    name     = "PostgreSQL Status Check"
#    args     = ["/bin/bash", "/etc/consul.d/check_postgres_status.sh"]
#    interval = "10s"
#  }
}

after consul validate and consul reload consul recognize this service and register this service.

Screenshot_20231112_175139
Screenshot_20231112_1751391076×771 44.2 KB

I deployed a simple Postgresql job on Nomad to verify connection and connect to my main database with psql command but it didn’t work, I even try this using curl 127.0.0.1:54321 to check my upstream port but it responds with curl: (56) Recv failure: Connection reset by peer.
My nomad job:

job "checker-job" {
  datacenters = ["dev"]

  group "checker-gp" {
    network {
      mode = "bridge"
      port "upstreams-from-port" {}
    }

    service {
      name = "upstreams-from"
      port = "upstreams-from-port"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "postgresql"
              local_bind_port  = 54321
              config {
                protocol = "tcp"
              }
            }
          }
        }
      }
    }

    task "checker-task" {
      driver = "docker"
      env {
        CONSUL_HTTP_TOKEN = "xyzxyzxyzxyzxyzxyzxyzxyzxyzxyz"
      }
      config {
        image = "postgres:14"
      }
      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

I tried this scenario with Redis but that didn’t work either, I don’t know what else should I do :frowning:

2

I resolve this issue using systemd service, but is it best practice?!

[Unit]
Description=Consul Envoy 4 postgresql
After=syslog.target network.target consul.service

[Service]
User=consul
ExecStart=/usr/bin/consul connect proxy -sidecar-for postgresql -token=xyzxyzxyzxyzxyzxyzxyzxyz
ExecStop=/bin/sleep 5
Restart=always

[Install]
WantedBy=multi-user.target