Can a service advertise multiple ports?

Nomad
1

I am setting up a Nomad/Consul/Terraform cluster for my company. We have created an application that exposes an HTTP endpoint for the main service. It also exposes an HTTP endpoint that displays Prometheus style metrics. Each of those endpoints are exposed through a different port.

In the network stanza I am able to specify multiple ports to expose for the group. Then in the task I am able to reference those ports and pass them to the configuration file for the application.

In the service stanza I am only able to specify one port to advertise to Consul. This means that in Prometheus, I have to manually define a target address in the static_configs for the metrics to be scraped from this application.

I would much prefer to take advantage of the consul_sd_configs in Prometheus. This application that I am setting up is only one of many and they all expose multiple ports (1 for app and 1 for metrics).

The following configuration works. I have commented out the dynamic port specifications since they do not seem to work.

job "awesome-api" {

  group "awesome-group" {

    network {
      port "http"    { host_network = "private" }
      port "metrics" {
        host_network = "private"
        static       = "9090"
      }
      //port "metrics" { host_network = "private" }
    }

    service {
      name    = "awesome-api"
      port    = "http"
      //port  = "metrics"
    }

    task "gateway" {
      driver = "exec"
      config {
        command = "${NOMAD_ALLOC_DIR}/api/app"
        args    = ["-config", "config.json"]
      }

      template {
        destination = "config.json"
        data        = <<-EOT
          {
            "http_address"      : "{{ env "NOMAD_IP_http" }}:{{ env "NOMAD_PORT_http" }}",
            "metrics_address"   : ":9090",
            //"metrics_address" : ":{{ env "NOMAD_PORT_metrics" }}",
          }
        EOT
      }
    }

  }
}

I saw this question asking about binding multiple ports. Though it seems a bit different, it seems like the answer may be the same, that Nomad cannot advertise multiple ports to Consul.

Can this be done in Nomad?

2 Likes
2

Also using the same standard metrics static port number for multiple services causes Placement Failures:

Dimension network: reserved port collision metrics=9090 exhausted on [x] nodes

Which means coming up with an arbitrary port number for each service that does not collide.

Our load-balancer is using Consul DNS, so using the metrics port as the one that is advertised is not a good alternative.

3

Since Nomad does not seem to be able to advertise multiple ports for a service, I found a workaround for my scenario. I am defining the second dynamic port and then saving it to the metadata. Consul is able to pass that metadata on to Prometheus where I am able to use relabel_configs to have Prometheus scrape the correct port dynamically while still using consul_sd_configs.

job "awesome-api" {

  group "awesome-group" {

    network {
      port "http"    { host_network = "private" }
      port "metrics" { host_network = "private" }
    }

    service {
      name    = "awesome-api"
      port    = "http"
    }

    meta {
      metrics-port = "${NOMAD_PORT_metrics}"
    }

    task "gateway" {
      driver = "exec"
      config {
        command = "${NOMAD_ALLOC_DIR}/api/app"
        args    = ["-config", "config.json"]
      }

      template {
        destination = "config.json"
        data        = <<-EOT
          {
            "http_address"    : "{{ env "NOMAD_IP_http" }}:{{ env "NOMAD_PORT_http" }}",
            "metrics_address" : ":{{ env "NOMAD_PORT_metrics" }}",
          }
        EOT
      }
    }

  }
}

Prometheus config snippet:

...

- job_name: 'all-services-consul-sd'
  metrics_path: "/"
  consul_sd_configs:
  - server: 'localhost:8500'

  relabel_configs:
    - source_labels: [__meta_consul_service_address,__meta_consul_service_metadata_metrics_port]
      separator: ';'
      regex: (.*);(\d{4,5})
      target_label:  '__address__'
      replacement: '$1:$2'
      action: 'replace'

...
4

@SunSparc,

I believe you can accomplish what you were looking for using a shared service name and discrete tags per service, that would enable you to get the metrics port or the http port depending on your need.

Here’s an example. Some elements are commented out to create a workload that I could start locally.

job "awesome-api" {
  datacenters = ["dc1"]
  type        = "service"
  
  group "awesome-group" {
    network {
      port "http"    {
        //host_network = "private"
      }
      port "metrics" {
        //host_network = "private"
      }
    }

    service {
      name = "awesome-api"
      port = "http"
      tags = ["http"]
    }

    service {
      name = "awesome-api"
      port = "metrics"
      tags = ["metrics"]
    }

    task "gateway" {
      driver = "exec"

      config {
//        command = "${NOMAD_ALLOC_DIR}/api/app"
//        args    = ["-config", "config.json"]
        command = "bash"
        args    = ["-c", "cat local/config.json; while true; do sleep 300; done"]
      }

      template {
        destination = "config.json"
        data        = <<-EOT
          {
            "http_address"    : "{{ env "NOMAD_IP_http" }}:{{ env "NOMAD_PORT_http" }}",
            "metrics_address" : ":{{ env "NOMAD_PORT_metrics" }}"
          }
        EOT
      }
    }
  }
}

Once you run this job, you will have 2 tags on the one service—http and metrics. You can reference them in a Consul DNS API query as http.awesome-api.service.consul and http.awesome-api.service.consul

Hopefully this can be useful for you. Best,
Charlie

3 Likes
5

Ah, run multiple services for the same task/group/job. I did not consider that. I think that is actually a much more elegant solution. Thanks for the tip, @angrycub.

6

Ok, I just tried the multiple services and it works so very well. Thank you again, @angrycub! Such a great way to start my Monday. :smiley:

1 Like