Vault server unable to connect to Consul

Vault
1

I am new to Consul/Vault/Nomad and I am attempting to set it up on my Raspberry Pi cluster so I can learn more about it. I am essentially using GitHub - chrisvanmeer/at-hashi-demo: Demonstration environment for the HashiCorp products as a guide with some modifications due to how my cluster setup differs from the VMs that repo is using. I got Vault configured and up and running. 2 of my 3 servers are connecting to Consul, but the third won’t connect. The server comes up, I can log into it via the CLI and the web UI, but it will not connect to Consul. I get a warning saying "service_registration.consul: reconcile unable to talk with Consul backend: error=“service registration failed: Unexpected response code: 403 (Permission denied: token with AccessorID ‘’ lacks permission ‘service:write’ on "vault")”

Vault config

# Enable UI for demo purposes
ui = true

# Cluster addresses
cluster_addr = "https://10.0.0.2:8201"
api_addr     = "https://10.0.0.2:8200"

# Consul storage backend
storage "consul" {
  address = "http://localhost:8500"
  path    = "vault/"
  token   = "<redacted>"
}

# TLS Listener
listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/opt/vault/tls/vault-cert.pem"
  tls_key_file       = "/opt/vault/tls/vault-key.pem"
  tls_client_ca_file = "/opt/vault/tls/vault-ca.pem"
}

Consul config

datacenter = "rivercrest"
data_dir = "/opt/consul"
encrypt = "<redacted>"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/rivercrest-server-consul-0.pem"
key_file = "/etc/consul.d/rivercrest-server-consul-0-key.pem"
verify_incoming = false
verify_outgoing = false
verify_server_hostname = false
retry_join = ["10.0.0.2", "10.0.0.3", "10.0.0.4"]
bind_addr = "10.0.0.2"
client_addr = "0.0.0.0"
ui = true

log_level = "info"
log_json = false
log_file = "/var/log/consul/"
log_rotate_duration = "86400s"
log_rotate_max_files = 7

acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens = {
    agent = "<redacted>"
  }
}

performance {
  raft_multiplier = 1
}

server = true
bootstrap_expect = 3
rejoin_after_leave = true

ports {
  grpc_tls = 8502
}

connect {
  enabled = true
}

My other 2 nodes (running on 10.0.0.3 and 10.0.0.4) are configured the exact same with the exact same tokens. They can connect just fine and show up in Consul, it is just this one node that won’t.

2

Nevermind, I solved this. Now, I am not sure why the other 2 were even connecting though. Apparently, the policy rules never got setup, at least not on the Consul server that the failing Vault server was connecting to. I forgot to check the other consul servers. But, when I updated it to include the correct rules like it should have, Vault started working.

1 Like
3

Glad you got it working and thanks for coming back to confirm. If you’re new to Vault, I also highly recommend running through these tutorials: