Failed to start Consul server: Failed to start Raft: permissions test failed

Hi,

currently we are facing an issue with consul on k8s. It gets unstable after a while and I am not sure what’s the cause of the issue but the following error is shown on the servers

[ERROR] agent: Error starting agent: error=“Failed to start Consul server: Failed to start Raft: permissions test failed: open /consul/data/raft/snapshots/permTest: interrupted system call”

kubernetes: 1.18.8
consul-helm: 0.25.0
consul-k8s: 0.18.1
consul: 1.8.5

We are also running another cluster with the same consul versions but on kubernetes 1.17.11 and its stable there, so my guess is that it is related to the kubernetes version.

Best regards,
Nico

Hi,
What environment are you running k8s in? I have not seen that error before. I can’t see many results for it in Google either. I’m guessing this is an issue with the underlying platform unfortunately.

Hi,

thanks for the response.

We are running consul on aks. I resolved the issue by removing the storageClass of the server which was set to azure file storage (kubernetes.io/azure-file).
As far as I understand this means if all consul servers go down, the state of the cluster and configuration is lost, are there any other drawbacks?

BR,
Nico

If that setting means that the disks are no longer persistent then yes, if the servers all go down then you will lose all your cluster data. I don’t think there are other drawbacks (although that’s a pretty big one!).