Can't see the nomad job labels in prometheus

Nomad
1

Hello,

I have configured vector on nomad to send the logs to Loki which is on the EC2 server.
But not able to see the labels in Prometheus for filtering the logs for example per job_name, namespace or node_name.

Here is my nomad script.

job "vector-loki" {
  datacenters = ["datacenter"]
  namespace   = "monitoring"
  type        = "system"
  update {
    min_healthy_time  = "10s"
    healthy_deadline  = "5m"
    progress_deadline = "10m"
    auto_revert       = true
  }
  group "vector" {
    count = 1
    restart {
      attempts = 3
      delay    = "15s"
      interval = "10m"
      mode     = "delay"
    }
    network {
      port "api" {
        to = 8686
      }
    }
    ephemeral_disk {
      size   = 500
      sticky = true
    }
    task "vector" {
      driver = "docker"
      config {
        image = "timberio/vector:0.24.2-debian"
        ports = ["api"]

        volumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ]
      }
      env {
        VECTOR_CONFIG          = "local/vector.toml"
        VECTOR_REQUIRE_HEALTHY = "true"
      }
      resources {
        memory = 100 # 100MB
      }
      template {
        destination     = "local/vector.toml"
        change_mode     = "signal"
        change_signal   = "SIGHUP"
        # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }}
        left_delimiter  = "[["
        right_delimiter = "]]"
        data            = <<EOH
          data_dir = "alloc/data/vector/"
          [api]
            enabled = true
            address = "0.0.0.0:8686"
            playground = true
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
          [sinks.loki]
            type = "loki"
            inputs = ["logs"]
            endpoint = "http://loki_address:3100"
            encoding.codec = "json"
            healthcheck.enabled = true
            labels.job = "{{ .job_name }}"
            labels.task = "{{ .task_name }}"
            labels.group = "{{ .task_group_name }}"
            labels.namespace = "{{ .namespace }}"
            labels.node = "{{ .node_name }}"
            labels.host = "{{ .host }}"
            labels.container_name = "{{ .container_name }}"
            remove_label_fields = true

        EOH
      }

      service {
        check {
          port     = "api"
          type     = "http"
          path     = "/health"
          interval = "30s"
          timeout  = "5s"
        }
      }
      kill_timeout = "30s"
    }
  }
}

I have added this extra_lables part in nomad.hcl config file on my 3 nodes.

plugin “docker” {
config {
extra_labels = [“job_name”, “task_group_name”, “task_name”, “namespace”, “node_name”]
volumes {
enabled = true
}
}
}

So after deploying to nomad I am not able to see the labels from which I can select and see appropriate job logs.

Please help me to fix this issue.

Thanks