Connection refused on sidecar proxy: can't get minimal example to work

Nomad
1

I’m trying to play around with Consul connect sidecars. I want traffic from the multitool to be proxied to httpbin. This job spec is what I came up with after reading just about everything I could fine about sidecars, but when I exec into the multitool container and run curl http://127.0.0.1:8080/get I get a connection refused.

job "httpbin" {
  datacenters = ["dc1"]
  type = "service"
  group "httpbin" {
    network {
      mode = "bridge"
    }
    service {
      name = "httpbin"
      port = 80 
      connect {
        sidecar_service {}
      }
    }
    task "httpbin" {
      driver = "docker"
      config {
        image = "kennethreitz/httpbin:latest"
      }
    }
  }
  group "multitool" {
    network {
      mode = "bridge"
    }
    service {
      name = "multitool"
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "httpbin"
              local_bind_port  = 8080
            }
          }
        }
      }
    }
    task "multitool" {
      driver = "docker"
      config {
        image = "wbitt/network-multitool:latest"
      }
      env {
        HTTPBIN_URL = "http://${NOMAD_UPSTREAM_ADDR_httpbin}"
      }
    }
  }
}
2

Hi @bradydean, I took a stab at fixing up the job file - looks like you just needed to setup some ports.

➜ curl localhost:8000
WBITT Network MultiTool (with NGINX) - 195f9a89d231 - 172.26.64.108 - HTTP: 80 , HTTPS: 443 . (Formerly praqma/network-multitool)

For reference, the way I debug this stuff is by looking at what addresses are being allocated to the allocation, and then exec-ing into the allocation to inspect what is actually listening to what address/port. Here’s the job file I ended up with:

job "httpbin" {
  datacenters = ["dc1"]
  type        = "service"

  group "httpbin" {
    network {
      mode = "bridge"
      port "http" {
        to = 8080
      }

    }
    service {
      name = "httpbin"
      port = "http"
      connect {
        sidecar_service {}
      }
    }
    task "httpbin" {
      driver = "docker"
      config {
        image = "kennethreitz/httpbin:latest"
        ports = ["http"]
      }
    }
  }

  group "multitool" {
    network {
      mode = "bridge"
      port "http" {
        static = 8000
        to     = 80
      }

    }
    service {
      name = "multitool"
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "httpbin"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "multitool" {
      driver = "docker"
      config {
        image = "wbitt/network-multitool:latest"
        ports = ["http"]
      }
      env {
        HTTPBIN_URL = "http://${NOMAD_UPSTREAM_ADDR_httpbin}"
      }
    }
  }
}

For debugging this type of stuff it helps to look at 2 things: the addresses being allocated to the allocation, e.g.

➜ nomad alloc status 37 | grep -A 3 'Allocation Addresses' 
Allocation Addresses (mode = "bridge"):
Label                     Dynamic  Address
*http                     yes      127.0.0.1:8000 -> 80
*connect-proxy-multitool  yes      127.0.0.1:22772 -> 22772

And then exec-ing into the allocation to inspect what is actually listening on which address, e.g.

➜ nomad alloc exec -task multitool 3726 /bin/bash
bash-5.1# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22772           0.0.0.0:*               LISTEN      -   (envoy downstream (unused))                
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1/nginx: master pro 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1/nginx: master pro 
tcp        0      0 127.0.0.2:19001         0.0.0.0:*               LISTEN      -   (envoy admin interface)                
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      -   (envoy upstream local bind)
3

Thanks for looking into this. As it turns out I didn’t have envoy installed… must have missed that step. Still having problems with it and consul w/tls (I just disabled tls for the moment, going to revisit that later I have a lot more to learn).

I have the httpbin/multitool job working, but this was just to get a minimal example working so I can have a reference for a larger mesh I’m working on with pgsql and some other services.

4

I’m not really sure what changed, but it isn’t working anymore. Instead of connection refused, I’m getting connection reset.

related: Intention doesn't seem to work: Traefik can't proxy requests to any services