Nomad with Vault Workload Identity permission denied on 1 client

Nomad
1

I’m not really sure the issue here. I had previously posted about how to manage tokens using vault and this setup in this post.

I was wondering how to add the token into the configuration and have it use a renewed token. The solution was the remove the token because you don’t need it. However, I now have a client that isn’t able to read secrets from vault and I’m not really sure where to start. Especially since the client has no vault token to speak of, so I don’t know why it would be working any different than other clients in the network.

The error I’m getting is this

vault.read(kv/data/services/bookstack/environment): Error making API request.

URL: GET http://vault.service.consul:8200/v1/kv/data/services/bookstack/environment
Code: 403. Errors:

* permission denied (retry attempt 9 after "1m0s"),

The server in question is a nomad, consul and vault client. The servers are on other machines.

The nomad configuration looks like this

data_dir      = "/var/lib/nomad"
plugin_dir    = "/etc/nomad/plugins"
bind_addr     = "0.0.0.0"
region        = "us"
datacenter    = "home"

log_level            = "INFO"
log_file             = "/var/log/nomad/service.log"
log_rotate_duration  = "24h"
log_rotate_max_files = 7
log_json             = true

advertise {
  http = "{{ GetPrivateIP }}:4646"
  rpc  = "{{ GetPrivateIP }}:4647"
  serf = "{{ GetPrivateIP }}:4648"
}

acl {
  enabled = false
}

server {
  enabled = false
}

leave_on_interrupt = true
leave_on_terminate = false

client {
  enabled = true
  server_join {
    retry_join = [(redacted)]
    retry_max = 3
    retry_interval = "15s"
  }

  drain_on_shutdown {
    deadline = "1h"
    force = false
    ignore_system_jobs = false
  }

  cni_path = "/opt/cni/bin"
  cni_config_dir = "/opt/cni/config"

  artifact {
    decompression_file_count_limit = 30000
  }

  host_volume "docker_sock" {
    path = "/var/run/docker.sock"
    read_only = true
  }

  host_network "tailscale" {
    interface = "tailscale0"
  }

  options = {
    "driver.raw_exec.enable" = "0"
  }
}

consul {
  address = "127.0.0.1:8500"
  token   = "(redacted)"
}

vault {
  enabled = true
  create_from_role = "nomad-server"
  address = "http://vault.service.consul:8200"
  default_identity {
    aud  = ["nomad"]
    ttl  = "1h"
  }
}