We have Consul server with 6 DCs. Primary DC runs on 1.15.2 version, and when we try to upgrade one of remote region cluster from 1.14.6 → 15.2 we get error:
failed to restore snapshot: error="failed to restore snapshot 7103-260348252-1682577557673: failed inserting acl token: missing value for index 'accessor'"
comparing raft snapshots we saw that it was not replicated from main DC, so we turned on:
"enable_token_persistence": true, "enable_token_replication": true
after that on remote DC we saw that we still have one legacy ACL token that has AccessorID with no value: (SecretID and Description changed intentionally so it won’t be exposed)
AccessorID: SecretID: xxxxxxx-xxxx-e40a-9fe8-99f6a6a3ccb3 Description: Acl1 Local: false Create Time: 0001-01-01 00:00:00 +0000 UTC Legacy: true
now we have no abilities to remove that tokens, as all token operations requires -id flag to be set and empty id is not a valid option.
maybe there is a possibility to force ACL token replication from main dc and/or somehow remove that token with minimal or no downtime?