I’m new to Nomad and was wanting to try it out on my server in a single node configuration. However I only wanted to deploy the ‘server’ part of the agent and the admin interface for a Traefik job onto my tailscale interface to avoid exposing it over the internet. It seems like this should be possible, but I’ve banged my head on the keyboard all night trying to get it to work.
My nomad.hcl file looks like this
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1
# Full configuration options can be found at https://developer.hashicorp.com/nomad/docs/configuration
data_dir = "/opt/nomad/data"
datacenter = "jobs"
bind_addr = "{{ GetInterfaceIP \"tailscale0\" }}"
# Server Configuration
server {
# license_path is required for Nomad Enterprise as of Nomad v1.1.1+
#license_path = "/etc/nomad.d/license.hclic"
enabled = true
bootstrap_expect = 1
}
acl {
enabled = true
}
# Client Configuration
client {
enabled = true
servers = ["127.0.0.1"]
host_network "public" {
cidr = "{{ GetInterfaceIP \"eth0\" }}/32"
reserved_ports = "22"
}
host_network "tailscale" {
cidr = "{{ GetInterfaceIP \"tailscale0\" }}/32"
}
}
My traefik job file looks like this
job "traefik" {
datacenters = ["jobs"]
type = "system"
group "traefik" {
count = 1
network {
mode = "host"
port "web" {
static = 80
host_network = "public"
}
port "websecure" {
static = 443
host_network = "public"
}
port "admin" {
static = 8080
host_network = "tailscale"
}
}
service {
name = "traefik-http"
provider = "nomad"
port = "web"
}
task "server" {
driver = "docker"
template {
data = <<EOH
NOMAD_TOKEN={{ with nomadVar "nomad/jobs" }}{{ .auth_token }}{{ end }}
EOH
destination = "secrets/file.env"
env = true
}
config {
image = "traefik:v3.2"
ports = ["admin", "web"]
args = [
"--api.dashboard=true",
"--api.insecure=true",
"--log.level=DEBUG",
"--entrypoints.web.address=:${NOMAD_PORT_web}",
"--entrypoints.websecure.address=:${NOMAD_PORT_websecure}",
"--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
"--providers.nomad=true",
"--providers.nomad.endpoint.address=http://${NOMAD_IP_admin}:4646", ### IP to your nomad server
"--providers.nomad.endpoint.token=${NOMAD_TOKEN}",
]
}
}
}
}
It’s been quite challenging to piece together the documentation for this kind of set up even when re-reading old forum posts. I’m sure I’m just not understanding something fundamental here, but would love some guidance!
I’m also open to other ideas/ways to ensure that the “management” side of these various tools is only accessible to my tail net and not the public internet.
Hi. So what is the issue? Looks ok. Are you able to run the job?
You used mode=host and told traefic to listen on all interfaces so it probably answers on all. I think either loose mode=host or tell explicitly to traefik to which ip to listen on.
Ah yes, I should have outlined my problem, apologies! So whenever I run the Traefik job I get a message that looks like this:
* Constraint "missing host network \"public\" for port \"web\"": 1 nodes excluded by filter
I’ve tried removing the mode="host" and also changing all of the ports to just be on the tailscale network just to remove a few variables. However when I change all of the ports to be on the tailnet I get the same error just with tailscale instead of public.
Even commenting out all of the host_network values inside of my traefik.nomad I still get the same error about 1 node being exlcuded by the filter.
Do I need to follow the steps at CNI plugins and bridge networking | Nomad | HashiCorp Developer to make this work? Also is there any way to verify that nomad is indeed binding to the tailscale0 interface? I ran sudo lsof -i :4646 and the output looks like:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nomad 3921027 root 17u IPv4 1586727155 0t0 TCP jobs.<tailnet.fqdn>:4646 (LISTEN)
So I think that means at least that part of the configuration is working, but would like to gut check that with folks.
Another thing that makes me think my Nomad configuration is wrong is that I have a test service defined as:
job "whoami" {
datacenters = ["jobs"]
type = "service"
group "demo" {
count = 1
network {
port "http" {
to = -1
}
}
service {
name = "whoami-demo"
port = "http"
provider = "nomad"
tags = [
"traefik.enable=true",
"traefik.http.routers.whoami.rule=Host(`whoami`)"
]
}
task "server" {
env {
WHOAMI_PORT_NUMBER = "${NOMAD_PORT_http}"
}
driver = "docker"
config {
image = "traefik/whoami"
ports = ["http"]
}
}
}
}
But I see it’s service by default using my public eth0 address instead of my tailscale0 address. I would have assumed since my bind_addr is specified as "{{ GetInterfaceIP \"tailscale0\" }}" this would be the default interface/ip used when creating services. Is that not the case?