# "Connection refused" to local Envoy sidecar

**URL:** <https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607>\
**Category:** Consul\
**Tags:** connect\
**Created:** [September 24, 2020, 2:50am UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607 "2020-09-24T02:50:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jfcantu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/jfcantu/32/8172_2.png) [@jfcantu](https://discuss.hashicorp.com/u/jfcantu)\
**Post date:** [September 24, 2020, 2:50am UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607/1 "2020-09-24T02:50:07Z")

</div>

I’m about to lose my mind.

My downstream service, for whatever reason, cannot connect Envoy to reach the upstream:

```auto
time="2020-09-23T22:46:47Z" level=info msg="Connecting to database at: bolt://127.0.0.1:7687"
time="2020-09-23T22:46:47Z" level=warning msg="Connection error: dial tcp 127.0.0.1:7687: connect: connection refused"

```

Here are my service definitions in Nomad:

Upstream:

```auto
    service {
      name = "${BASE}-bolt-internal"
      port = "bolt"
      connect {
        sidecar_service {}
      }
    }

```

Downstream:

```auto
    service {
      name = "${BASE}-api"
      port = "web"
      check {
        type = "http"
        port = "web"
        path = "/"
        interval = "60s"
        timeout = "10s"
      }
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "jnexus-database-group-bolt-internal"
              local_bind_port = 7687
            }
          }
        }
      }
    }

```

Sidecar logs from the downstream don’t show anything abnormal. The listener starts and then the sidecar gets terminated because the main process dies.

```auto
[2020-09-24 02:27:32.674][1][info][main] [source/server/server.cc:500] all clusters initialized. initializing init manager
[2020-09-24 02:27:32.681][1][info][upstream] [source/server/lds_api.cc:60] lds: add/update listener 'public_listener:0.0.0.0:31919'
[2020-09-24 02:27:32.682][1][info][upstream] [source/server/lds_api.cc:60] lds: add/update listener 'jnexus-database-group-bolt-internal:127.0.0.1:7687'
[2020-09-24 02:27:32.682][1][info][config] [source/server/listener_manager_impl.cc:761] all dependencies initialized. starting workers
[2020-09-24 02:28:15.264][1][warning][main] [source/server/server.cc:468] caught SIGINT
[2020-09-24 02:28:15.264][1][info][main] [source/server/server.cc:567] shutting down server instance
[2020-09-24 02:28:15.264][1][info][main] [source/server/server.cc:521] main dispatch loop exited
[2020-09-24 02:28:15.265][1][info][main] [source/server/server.cc:560] exiting

```

---

<div class="post-metadata">

**Author:** ![martinkingtw](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/martinkingtw/32/11886_2.png) [@martinkingtw](https://discuss.hashicorp.com/u/martinkingtw)\
**Post date:** [September 24, 2020, 3:32am UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607/2 "2020-09-24T03:32:10Z")

</div>

Are you sure that ${BASE} would resolve to `jnexus-database-group`?  
Why not try a static name first, to see whether it will work or not?

---

<div class="post-metadata">

**Author:** ![jfcantu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/jfcantu/32/8172_2.png) [@jfcantu](https://discuss.hashicorp.com/u/jfcantu)\
**Post date:** [September 24, 2020, 4:31am UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607/3 "2020-09-24T04:31:12Z")

</div>

Nope, same problem.

I should also mention that I have Consul ACLs enabled and intentions configured between the services.

---

<div class="post-metadata">

**Author:** ![martinkingtw](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/martinkingtw/32/11886_2.png) [@martinkingtw](https://discuss.hashicorp.com/u/martinkingtw)\
**Post date:** [September 24, 2020, 6:06am UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607/4 "2020-09-24T06:06:35Z")

</div>

Did you give nomad enough access to register the services to consul?

You have to create token for nomad server and client.

Example

```auto
agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "write"
}

acl = "write"

```

Read it here.

> **[Secure Nomad Jobs with Consul Service Mesh | Nomad - HashiCorp Learn](https://learn.hashicorp.com/tutorials/nomad/consul-service-mesh)**
>
> Configure Nomad for an ACL-enabled Consul cluster and run a sample job
> that uses Consul service mesh.

Alternatively, are you 100% sure that the shutdown is caused by not finding services on port 7689? It might be some other problems causing SIGINT.

---

<div class="post-metadata">

**Author:** ![jfcantu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/jfcantu/32/8172_2.png) [@jfcantu](https://discuss.hashicorp.com/u/jfcantu)\
**Post date:** [September 24, 2020, 1:44pm UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607/5 "2020-09-24T13:44:28Z")

</div>

Yes. The upstream shows as registered in Consul and passes health checks, and Nomad shows that the sidecar is getting terminated because the task it’s attached to has failed.

---

<div class="post-metadata">

**Author:** ![jfcantu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/jfcantu/32/8172_2.png) [@jfcantu](https://discuss.hashicorp.com/u/jfcantu)\
**Post date:** [September 26, 2020, 4:06am UTC](https://discuss.hashicorp.com/t/connection-refused-to-local-envoy-sidecar/14607/6 "2020-09-26T04:06:16Z")

</div>

Boy, was this an adventure to figure out.

TL;DR - Don’t mix Nomad’s and Docker’s `bridge` modes - they’re different.

I had followed [this guide](https://medium.com/zendesk-engineering/making-docker-and-consul-get-along-5fceda1d52b9), which details how to configure Consul DNS to work inside Docker containers by creating a dummy network interface that can passed to the Docker `--dns` option.

The `--dns` option works just fine with Docker’s `bridge` mode. It does _not_ work with Nomad’s `bridge` mode - because Nomad’s `bridge` mode sets the Docker networking mode to `none` so the CNI plugin can be used instead. If you try and specify `config.dns_servers` for a Nomad task that’s operating in Nomad `bridge` mode, you’ll get an error from Docker saying that `dns_servers` is incompatible with your networking mode.

So of course, when I got this error, I simply set Docker’s `config.networking_mode` to `bridge` for the task, the error disappeared, and I went on my merry way.

Unbeknownst to me, this had switched me away from using the CNI plugin, and I was now in Docker’s `bridge` mode in the default `bridge` network.

And, well - in that mode, `127.0.0.1` is not shared between containers… so when my service tried to reach the sidecar, it was trying to connect back to its own container, where Envoy isn’t running.