Consul Template error automatic rotate gossip key

Consul
1

Hello there, Im new to vhault and consul and i have been trying out this https://learn.hashicorp.com/tutorials/consul/vault-kv-consul-secure-gossip?in=consul/vault-secure for hashicorps own website. Alot of it has been successful for me. Its at the last part in which Im am getting the error. The switching the gossip keys its that a really long time and its not consistent. Could someone please explain the error Im seeing and how can i fix it. Thank you

==> Installing new gossip encryption key...
error: Unexpected response code: 500 (4 errors occurred:
	* WAN error: 1/1 nodes reported failure
	* vishnu-Latitude-5521.dc1: No keyring to modify (encryption not enabled)
	* dc1 (LAN) error: 1/1 nodes reported failure
	* <Node Name>: No keyring to modify (encryption not enabled))
==> Changing primary gossip encryption key...
error: Unexpected response code: 500 (4 errors occurred:
	* WAN error: 1/1 nodes reported failure
	* vishnu-Latitude-5521.dc1: No keyring to modify (encryption not enabled)
	* dc1 (LAN) error: 1/1 nodes reported failure
	* <Node Name>: No keyring to modify (encryption not enabled))
jq: error (at <stdin>:1): Cannot iterate over number (4)
parse error: Invalid numeric literal at line 1, column 9
2

Hi @vishnuj,

Welcome to the HashiCorp Forums!

From the error message, it looks like the agents in your cluster don’t have Gossip Encryption enabled.

No keyring to modify (encryption not enabled)

The pre-requisite in the doc says that you should have a Consul cluster that already has gossip encryption enabled.

ref: Automatically Rotate Gossip Encryption Keys Secured in Vault | Consul - HashiCorp Learn

Considering that you are running a -dev agent, use the -encrypt arg to enable gossip encryption and re-run consul-template.

eg:

$ consul agent -dev -encrypt $(consul keyring)

I hope this helps.

3

Hello @Ranjandas , thank you so much for your reply. The solution you suggested work briefly for me. The gossip key was being updated correctly without and error.

From there, I tried to play around with the gossip.key.tpl and see if i can read multiple key which didnt result in anything good. So i decided to revert back to the version that was working fine for me and now this error is shown.

==> Installing new gossip encryption key...
error: Unexpected response code: 500 (2 errors occurred:
	* WAN error: illegal base64 data at input byte 6
	* dc1 (LAN) error: illegal base64 data at input byte 6)
==> Changing primary gossip encryption key...
error: Unexpected response code: 500 (2 errors occurred:
	* WAN error: illegal base64 data at input byte 6
	* dc1 (LAN) error: illegal base64 data at input byte 6)
==> Removing gossip encryption key...
error: Unexpected response code: 500 (4 errors occurred:
	* WAN error: 1/1 nodes reported failure
	* vishnu-Latitude-5521.dc1: Removing the primary key is not allowed
	* dc1 (LAN) error: 1/1 nodes reported failure
	* vishnu-Latitude-5521: Removing the primary key is not allowed)
2022-09-14T17:08:59.630+0800 [ERR] (cli) 1 error occurred:
	* failed to execute command "[\"/opt/rotate_key.sh\"]" from "/opt/consul/templates/gossip.key.tpl" => "/opt/consul/gossip/gossip.key": child: command exited with a non-zero exit status:

    /opt/rotate_key.sh

This is assumed to be a failure. Please ensure the command
exits with a zero exit status.

I even tried deleting the whole file and creating a new gossip.key.tpl but nothing seems to work. I also tried restarting consul agent but still nothing works. Please help and thank you.