Duplicate Nomad services appearing in templates. How do I remove them?

Nomad
1

Hi there.

I’m using nomad 1.6.1 and and I have jobfile like so:

job "rabbitmq_server" {
  datacenters = ["dc1"]

  type = "service"

  group "rabbit" {

    count = 1

    network {
      mode = "host"

      port "rabbit" {
        static       = 5672
        host_network = "my-network"
      }

      // used by some CLI tools. See below for more:
      // https://rabbitmq.com/networking.html
      port "rabbit-epmd" {
        static       = 4369
        host_network = "my-network"
      }
      
      // used for management 
      // https://rabbitmq.com/management.html
      port "rabbit-dashboard" {
        static        = 15672
        host_network = "my-network"
      }
      // used for prometheus metrics
      // https://www.rabbitmq.com/prometheus.html
      port "rabbit-prometheus" {
        static        = 15692
        host_network = "my-network"
      }
      

    }

    task "rabbit" {
      driver = "docker"

      template {
        data        = file("./nomad/local/rabbitmq.enabled.plugins.tpl")
        destination = "/etc/rabbitmq/enabled_plugins"
      }

      config {
        image = "rabbitmq:3.12-management"
        ports = ["rabbit", "rabbit-dashboard", "rabbit-epmd", "rabbit-prometheus"]
        hostname = "greenrabbit"
      }

      service {
        provider = "nomad"
        name = "rabbit-dashboard"
        port = "rabbit-dashboard"
      }

      service {
        provider = "nomad"
        name = "rabbitmq"
        port = "rabbit"

        // check every 10s that the we can connect over http
        // Using tcp messages cause errors in the rabbitmq logs
        check {
          type     = "http"
          path     = "/"
          port     = "rabbit-dashboard"
          interval = "10s"
          timeout  = "2s"
        }
      }
    
      resources {
        cpu    = 1000 # 1000 MHz
        memory = 1024 # 1024 GB
      }

      constraint {
        attribute = "${attr.unique.hostname}"
        value     = "db2.greenweb.org"
      }
    }
  }
}

When I template out to refer to these seevices in another job, I’m seeing the same port and ip address being declared twice in templates, and I’m not sure how to remove it.

I’m using caddy to refer to a service like this:


nomad.my.org {
	reverse_proxy 10.0.0.2:4646
}

rabbit.my.org {
	{{ range nomadService "rabbit-dashboard" }}
	reverse_proxy {{.Address}}:{{.Port}}
	{{ end }}
}

And I’m getting the reverse proxy line twice in the template output like so:

nomad.my.org {
	reverse_proxy 10.0.0.2:4646
}

rabbit.my.org {
	
	reverse_proxy 10.0.0.8:15672
	
	reverse_proxy 10.0.0.8:15672
	
}

I think this is down to there being two allocations of a service, and I can see this when I navigate through the UI, but the second of the two allocations is unreachable in the UI. It also shows up when I examine a service.

When I call this:

nomad service info rabbitmq

I get this output, listing the two allocations serving it from the same nomad client:

Job ID           Address        Tags  Node ID   Alloc ID
rabbitmq_server  10.0.0.8:5672  []    2113aa11  d3f111c4
rabbitmq_server  10.0.0.8:5672  []    2113aa11  f7520c50

I think this is related to failed deploy, or something similar before, but it’s not obvious how to remove it.

I see this command here:

But the warnings make me hesitant to use it, as the docs refer to a registration_id, not an allocation, and I’m not sure what a registration is in this context.

The example makes me think it’s not supposed to be used with allocation ids as none of my allocations id folow this pattern either:

 nomad service delete example-cache-redis _nomad-task-a831f7f2-4c01-39dc-c742-f2b8ca178a49-redis-example-cache-redis-db

How do I clear it out?