Consul Connect Envoy without Docker

Nomad
1

I have spend the last couple weeks learning Consul Connect and trying to use it with Nomad. However, nearly all examples I’ve seen involve using the Docker driver. Are there any good examples or walkthroughs of setting up Consul Connect with Envoy in Nomad with the raw_exec or exec drivers?

I have gotten Consul Connect with the built-in proxy to work by explicitly defining the proxies as separate tasks within Nomad jobs. I have also gotten Consul Connect with Envoy and adjusted this walkthrough to work without Docker, but can’t seem to figure out how to wrap it in Nomad:

And of course there is this guide, But I can’t quite figure out how to extrapolate this walkthrough to a non-docker solution:

Thanks in advance!

3 Likes
2

Hi,

I have not tested this, but it looks like you should be able to use the sidecar_task stanza to define that the sidecar should use the exec or raw_exec task driver.

The Envoy binary will need to be preinstalled on the server, or downloaded using artifact. GetEnvoy.io provides Envoy binaries for a number of operating system distributions, and is probably the easiest way to obtain a binary outside of exporting it from the Docker image.

Hope this helps.

1 Like
3

@jcouvret I’ve got a couple of examples using Connect with exec and java drivers:

If you’re asking about having a non-Docker Envoy sidecar, then @blake has the right approach that you’ll want to use the sidecar_task. I’ll admit that’s not an approach we’ve heavily tested to date but I’d love to see if you manage to get it working.

1 Like
4

I’d love to be able to have services deployed by nomad (raw_exec) to be able to leverage connect+envoy without docker. I was able to easily get consul+connect+envoy running with envoy standalone… But I really like nomad and would like to continue using it, so I’m at an impasse. Right now I’m considering some mechanism to dump a consul config file, loading the envoy proxy as a separate task, and linking them together… but with little experience with nomad, I’m finding it challenging. Any advice would be appreciated.

1 Like
5

Hi everybody,

in our context we are trying to deploy jobs with sidecars in mode “no-docker”, so with “exec” driver (through sidecar_task stanza). We found that we can do it and it works flawlessly when we use “docker” as driver of the task the sidecar is next to.

But when we use a “java” or “exec” driver in the task, then we have some problems, and envoy logs are:

[2020-07-28 17:03:00.148][13055][info][main] [external/envoy/source/server/server.cc:255] initializing epoch 0 (hot restart version=11.104)
[2020-07-28 17:03:00.148][13055][info][main] [external/envoy/source/server/server.cc:257] statically linked extensions:
...
...
[2020-07-28 17:03:00.156][13055][info][main] [external/envoy/source/server/server.cc:340] admin address: 127.0.0.1:19001
[2020-07-28 17:03:00.156][13055][critical][main] [external/envoy/source/server/server.cc:95] error initializing configuration '/secrets/envoy_bootstrap.json': cannot bind '127.0.0.1:19001': Cannot assign requested address
[2020-07-28 17:03:00.156][13055][info][main] [external/envoy/source/server/server.cc:606] exiting
cannot bind '127.0.0.1:19001': Cannot assign requested address

In the job we create a network at group level in bridge mode. Could that, along with “non-docker” task driver, be the problem?

Thank you

Francesco

1 Like
6 
[2020-07-28 17:03:00.156][13055][info][main] [external/envoy/source/server/server.cc:606] exiting
cannot bind '127.0.0.1:19001': Cannot assign requested address

Turns out to be a problem with missing loopback interface. I don’t know if this is an intended “feature” but any process (as far as I could test) started with exec isolation, only has eth0 bridged interface visible. Running ifup lo as a prestart hook makes the symptom go away.

2 Likes
7

Thank you @gfdsa - that was nice find. I would be curious if anyone here from HashiCorp might be able to let us know if this is expected behavior or a bug.

To help anyone else out searching how to do this, this is an example of a portion of the countdash job with no docker required. Based on my testing, it appears that Nomad tasks utilizing Consul Connect cannot be raw_exec. I would be interested if anyone else figures out how to use raw_exec.

job "countdash" {
  datacenters = ["dc1"]

  group "api" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"

      connect {
        sidecar_service {}
        sidecar_task {
          driver = "exec"
          config {
            command = "/bin/envoy"
            args  = [ "-c", "${NOMAD_SECRETS_DIR}/envoy_bootstrap.json",
                    "-l", "${meta.connect.log_level}" ]
          }
          resources {
            cpu    = 50
            memory = 50
          }
        }
      }
    }

    task "init_lo" {
      driver = "raw_exec"
      user = "root"
      config {
        command = "/usr/sbin/ifup"
        args = ["lo"]
      }
      lifecycle {
        hook    = "prestart"
      }
    }

    task "web" {
      driver = "exec"
      env {
        PORT = "9001"
      }
      config {
        command = "/bin/counting-service"
      }
    }
  }
}
2 Likes
8

Hypothetically there can be use cases when a sidecar proxy is running without isolation. But I wouldn’t feel safe with having no way of authentication of the non-native client running on the same localhost. And the port numbers… Just my two cents.

9

I’m very late to the party but there’s a confirmed bug report for it now: When running container via containerd-driver plugin in the bridge mode, there is no ip address(127.0.0.1/8) in the lo network · Issue #10014 · hashicorp/nomad · GitHub
(Still no fix for it though, unfortunately.)

1 Like