I’m having a very odd issue, where after a
vault operator rekey operation, the nodes in the HA cluster disagree.
I’ve attempted to
-cancel the operation and restart it, but nothing has worked. When we attempted the rekey, we used 9 shares with 4 threshold, and though 5 people put their keys in (current threshold) the rekey operation simply did not occur, and the progress of the rekey operation with the same nonce reset itself to 0/5 after 5 people put in their keys.
Here’s a screenshot of the same status command directly to the lead node (not through the LB) where the Rekey Progress inexplicably resets itself.
And a screenshot of the status of each node in the HA cluster:
As you can see, the nodes in the same cluster disagree about their shares and threshold values. And none of them match what we were attempting, which was 9 shares and 4 threshold.
I did a quick search of the logs on the /sys/rekey endpoints and the only thing that shows up in the logs are the calls to /sys/rekey/backup when I attempted to see what the current keys were.
When I started the re-key operation, my command was similar to the following:
vault operator rekey \ -init \ -key-shares=9 \ -key-threshold=4 \ -pgp-keys="key1.asc,key2.asc,key3.asc,key4.asc,key5.asc,key6.asc,key7.asc,key8.asc,key9.asc" \ -backup
Does anyone have any clues as to what I could attempt here?