Unexpected response code: 500 (No cluster leader)

Consul
1

This is in reference to a GitHub question posted here: https://github.com/hashicorp/consul/issues/7927

I setup vault with consul cluster in AWS using Terraform. they set up ran for a month and all the sudden it starts giving me an error.
500 (The backend responded with an error)
Consul 1.7.2
Vault v1.4.0

2 vault servers
2 consul servers.

initial diagnosis suggests that no leader is selected.
##consul operator raft list-peers -stale
Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 500 (No cluster leader)

consul members

Node Address Status Type Build Protocol DC Segment
i-073a14454e1789f6f 172.31.2.238:8301 alive server 1.7.2 2 eu-west-3
i-0ca6511481c563e8e 172.31.42.142:8301 alive server 1.7.2 2 eu-west-3
i-000e40b72ddc95924 172.31.30.128:8301 alive client 1.7.2 2 eu-west-3
i-0ffc15ccd24faf9af 172.31.12.115:8301 alive client 1.7.2 2 eu-west-3

web browser shows below

image
image1449×264 14.6 KB

with console

image
image1910×290 41.2 KB

Network tab

image
image1914×298 33.2 KB

I restart all the servers that didn’t help

2

Hi @yagashiraevil ,

Thanks for posting, and sorry you’re running into issues!

I noticed that you are using an even number of Consul servers. In our Consul production guide, and the Vault with Consul learn guide we generally recommend 3-5 servers to provide quorum. It could be that your hosts are in a split-brain scenario.

  • Can you please post your Consul and Vault server configs, with the sensitive bits removed?
  • Can you please try running the curl commands found here to output the leader? Do they match your error?
  • What terraform module did you use to set up Consul?
  • Were there any recent changes to the module, or a recent terraform apply that may have happened?

I hope this helps guide your questions.