Update service parameters on the flly

Consul
1

Is there a way to update service parameters (tags or meta) on the fly. So that I can, for example, add or remove some tag for a given service while it is working.

I see that I can do a register request to achieve this: get service description, do updates to tags and register this “patched” description. But this approach has some tricky edge cases like the service is deregitered in between “get current description” and “register with updated description”, and so I’ll have a phantom service.

Could you, please, guide me with what would be an approach handle it.

I’m also thinking of using KV for this, let’s say I have a meta: “dynamic_subject_key” => “key_path_to_what would_be_changing_from_time_to_time”. And thus having a constant service description service-clients would be able to get something that is changed from time to time.

2

Hi @ngrodzitski,

It is possible to centrally update tags for individual services instances through the catalog API endpoint. You must first set EnableTagOverride to true in the service definition (enable_tag_override if using JSON syntax) when registering the service instance with the local agent either through the /agent/service/register API or the CLI.

The tag override feature is documented here https://www.consul.io/docs/discovery/services#enable-tag-override-and-anti-entropy.

As you noted, a service’s tags can be updated using the /v1/catalog/register endpoint. If you want to avoid the scenario where a service has been updated or deleted since the previous GET operation, its better to update the tags using the transactions API and the cas (check-and-set) verb. This will only update the resource if it the ModifyIndex matches the provided value in the update payload.

Example

  1. Register two services instances for the logical service web, instance web1 and instance web2.

    The web1 instance does not permit overriding tags from the catalog endpoint. The web2 instance does allow tags to be overridden.

    {
  "service": {
    "id": "web1",
    "name": "web",
    "port": 80,
    "enable_tag_override": false
  }
}

    {
  "service": {
    "id": "web2",
    "name": "web",
    "port": 80,
    "enable_tag_override": true
  }
}

    $ consul services register web1.json
Registered service: web
$ consul services register web2.json
Registered service: web

  2. Query the service catalog to confirm the web service has been registered. The output shows there are no tag associated with the logical service.

    $ curl localhost: 8500/v1/catalog/services
{
  "consul": [],
  "web": []
}

  3. Query /catalog/service/web to see the details of the registered instances. This will return all instances of the web service.

    $ curl localhost: 8500/v1/catalog/service/web
[
  {
    "ID": "83fa7111-8377-b1f1-9201-d2c5033faeb3",
    "Node": "blake-C02YX6QSLVCG",
    "Address": "127.0.0.1",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "127.0.0.1",
      "lan_ipv4": "127.0.0.1",
      "wan": "127.0.0.1",
      "wan_ipv4": "127.0.0.1"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceKind": "",
    "ServiceID": "web1",
    "ServiceName": "web",
    "ServiceTags": [],
    "ServiceAddress": "",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 80,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 59,
    "ModifyIndex": 59
  },
  {
    "ID": "83fa7111-8377-b1f1-9201-d2c5033faeb3",
    "Node": "blake-C02YX6QSLVCG",
    "Address": "127.0.0.1",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "127.0.0.1",
      "lan_ipv4": "127.0.0.1",
      "wan": "127.0.0.1",
      "wan_ipv4": "127.0.0.1"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceKind": "",
    "ServiceID": "web2",
    "ServiceName": "web",
    "ServiceTags": [],
    "ServiceAddress": "",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 80,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": true,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 63,
    "ModifyIndex": 81
  }
]

  4. To restrict the output to service instance(s) which permit having their tags modified, use a filter to instances with ServiceEnableTagOverride equal to true.

    $ curl --get localhost: 8500/v1/catalog/service/web --data-urlencode "filter=ServiceEnableTagOverride == true"
[
  {
    "ID": "83fa7111-8377-b1f1-9201-d2c5033faeb3",
    "Node": "blake-C02YX6QSLVCG",
    "Address": "127.0.0.1",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "127.0.0.1",
      "lan_ipv4": "127.0.0.1",
      "wan": "127.0.0.1",
      "wan_ipv4": "127.0.0.1"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceKind": "",
    "ServiceID": "web2",
    "ServiceName": "web",
    "ServiceTags": [],
    "ServiceAddress": "",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 80,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": true,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 63,
    "ModifyIndex": 81
  }
]

  5. The next step is to pass the API response output to a script that will add the desired tags to the eligible services. The script should produce a JSON payload that can be used by the /v1/txn API endpoint to update the services in the catalog.

    The payload syntax for updating a service in a transaction is as follows.

    [
  {
    "Service": {
      "Verb": "cas",
      "Node": "<node_name>",
      "Service": "<service details, including ModifyIndex>"
    }
  }
]

    For this example, use jq to generate the transaction update payload from the response output of the /v1/catalog/service/:service query. Save the following text into a file named service-transform.jq.

    def nest_service_params:
    # Select all objects with keys that start with the the name 'Service'
    # Remove 'Service' prefix from key names
    with_entries(select(.key | startswith("Service")) | .key = (.key | sub("^Service"; "")))
# End function
;

def create_svc_txn:
  # Append tags to service entry
  .ServiceTags = [
  "new_tag"
]

  # Rename 'ServiceName' to 'ServiceService'
  # Will be transformed to 'Service.Service later'
  | . + {ServiceService: .ServiceName
}
  | del(.ServiceName)

  | nest_service_params as $nested_params

  # Create transaction to update services using check-and-set option.
  # If service has been modified since our last read, or no longer exists, the
  # transaction will fail.
  | {
      Service: {
        Verb: "cas",
        Node: .Node,

        # Nest service-related parameters under 'Service' key
        # Add ModfyIndex key
        Service: ($nested_params + {ModifyIndex: .ModifyIndex
    })
  }
}
# End function
;

# Convert /v1/catalog/service/:service response to a suitable
# array for /v1/txn
. | map(create_svc_txn)

  6. Pipe the output of the previous curl command to the jq filter. This will result in the following payload being produced.

    [
  {
    "Service": {
      "Verb": "cas",
      "Node": "blake-C02YX6QSLVCG",
      "Service": {
        "Kind": "",
        "ID": "web2",
        "Tags": [
          "new_tag"
        ],
        "Address": "",
        "Weights": {
          "Passing": 1,
          "Warning": 1
        },
        "Meta": {},
        "Port": 80,
        "SocketPath": "",
        "EnableTagOverride": true,
        "Proxy": {
          "Mode": "",
          "MeshGateway": {},
          "Expose": {}
        },
        "Connect": {},
        "Service": "web",
        "ModifyIndex": 583
      }
    }
  }
]

  7. Update the tags in the service catalog by tying the previous steps together with one last command to PUT the transaction update payload to the API endpoint.

    The final set of commands will perform the following actions:

    1. Obtain the latest service instance information from the catalog using curl.
    2. Use a jq filter to construct the transaction API payload that will be used to update the tags.
    3. Update the tags using curl.
    $ curl --silent --get localhost:8500/v1/catalog/service/web --data-urlencode "filter=ServiceEnableTagOverride == true" | \
jq --from-file service-transform.jq --compact-output | \
curl --request PUT localhost:8500/v1/txn --data @-

    The API should return a successful response if the tags have been updated.

    {
  "Results": [
    {
      "Service": {
        "ID": "web2",
        "Service": "web",
        "Tags": [
          "new_tag"
        ],
        "Address": "",
        "Meta": {},
        "Port": 80,
        "Weights": {
          "Passing": 1,
          "Warning": 1
        },
        "EnableTagOverride": true,
        "Proxy": {
          "Mode": "",
          "MeshGateway": {},
          "Expose": {}
        },
        "Connect": {},
        "CreateIndex": 388,
        "ModifyIndex": 580
      }
    }
  ],
  "Errors": null
}

  8. Query the /v1/agent/service endpoint for the web service instances. You will see the new_tag value present in the list of service tags for the web2 service instance, not web1.

    $ curl --get localhost: 8500/v1/catalog/service/web --data-urlencode "filter=Service==web"         
{
  "web1": {
    "ID": "web1",
    "Service": "web",
    "Tags": [],
    "Meta": {},
    "Port": 80,
    "Address": "",
    "SocketPath": "",
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "web2": {
    "ID": "web2",
    "Service": "web",
    "Tags": [
      "new_tag"
    ],
    "Meta": {},
    "Port": 80,
    "Address": "",
    "SocketPath": "",
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": true,
    "Datacenter": "dc1"
  }
}