Consul AccessorID is empty in one of remote clusters

Consul
1

We have Consul server with 6 DCs. Primary DC runs on 1.15.2 version, and when we try to upgrade one of remote region cluster from 1.14.6 → 15.2 we get error:

failed to restore snapshot: error="failed to restore snapshot 7103-260348252-1682577557673: failed inserting acl token: missing value for index 'accessor'"

comparing raft snapshots we saw that it was not replicated from main DC, so we turned on:

        "enable_token_persistence": true,
        "enable_token_replication": true

after that on remote DC we saw that we still have one legacy ACL token that has AccessorID with no value: (SecretID and Description changed intentionally so it won’t be exposed)

AccessorID:
SecretID:         xxxxxxx-xxxx-e40a-9fe8-99f6a6a3ccb3
Description:      Acl1 
Local:            false
Create Time:      0001-01-01 00:00:00 +0000 UTC
Legacy:           true

now we have no abilities to remove that tokens, as all token operations requires -id flag to be set and empty id is not a valid option.
maybe there is a possibility to force ACL token replication from main dc and/or somehow remove that token with minimal or no downtime?

2

Oh dear.

The legacy ACL system was completely removed in Consul 1.11, so for you to have a cluster at 1.14 that still has a legacy token defined, indicates something has gone very wrong.

Since the legacy ACL system has been removed, so has all the code meant to upgrade and manipulate legacy tokens.

It seems you are in uncharted territory :-/

The only ideas that occur to me right now are:

  • Total outage for that DC, destroy the Consul servers’ state and re-set-up Consul from scratch.

  • Create a custom patched version of Consul 1.14, which will automatically delete any legacy tokens it sees on becoming a leader, and run that temporarily in the problem DC.