Nomad root Client and Podman rootless socket

Nomad
1

Hello,

I’ve struggled a lot with podman rootless and Nomad, but in the end I’ve managed to make it work. The only thing is that I was running my Client as “hashi” user with UID/GID 2000 which is the same as my podman socket. Everything on a single server. Then I hit a wall when I tried to add a bridge network to use consul connect due to lack of permission.
So I tried to switch to Nomad Server service as hashi without client stanza + Nomad Client service as root:

** Server **

data_dir = "/var/opt/nomad"
server {
  enabled = true
  bootstrap_expect = 1
}
vault {
  enabled = true
  address = "http://localhost:8200"
  token = "<token>"
  task_token_ttl = "72h"
  create_from_role = "nomad-cluster"
}

consul {
    token = "<token>"
    address = "server.dc1.consul:8501"
    ssl       = true
    ca_file = "/mnt/tank/config/consul.d/consul-agent-ca.pem"
    cert_file = "/mnt/tank/config/consul.d/dc1-server-consul-0.pem"
    key_file = "/mnt/tank/config/consul.d/dc1-server-consul-0-key.pem"
}

Client

plugin_dir = "/var/opt/nomad/plugins"
data_dir = "/var/opt/nomad-client"

ports {
  http = 5646
  rpc  = 5647
  serf = 5648
}
client {
  enabled = true
}

plugin "nomad-driver-podman" {
  config {
    socket_path="unix://run/user/2000/podman/podman.sock"
    volumes {

      enabled      = true
      selinuxlabel = "z"
    }
  }
}

vault {
  enabled = true
  address = "http://localhost:8200"
  token = "<token>"
  task_token_ttl = "72h"
  create_from_role = "nomad-cluster"
}

consul {
    token = "<token>"
    address = "server.dc1.consul:8501"
    ssl       = true
    ca_file = "/mnt/tank/config/consul.d/consul-agent-ca.pem"
    cert_file = "/mnt/tank/config/consul.d/dc1-server-consul-0.pem"
    key_file = "/mnt/tank/config/consul.d/dc1-server-consul-0-key.pem"
}

nomad.service is running as hashi/hashi
nomad-client.service is running as root/root
podman.socket is running as user service for hashi enabling podman.service

When I try to run jobs now it says
Feb 03 12:31:52 nomad.local podman[16163]: [conmon:e]: Failed to open log file Permission denied

I also had a lot of issues with alloc folders in client var folder which I’m not sure I managed to fix properly. It seems to me that since my client is root and my socket is hashi user, it struggles to properly write logs and such. Can I set somewhere in my client to use user hashi from a root service?

PS: some of my config is there https://github.com/TecHunter/nomad-consul-vault but I need to stabilize the root client first before pushing latest jobs and host fixes