Vault with keepalived vault not listening to virtual ip

Vault
1

hello, Im running a 3 node cluster for autounseal and a 3 node cluster for ssh signing certs. I have keepalived running for the autounseal cluster i the tracking script will poll the /v1/sys/health endpoint and check its status.
i can ping the vip but when i try to curl sys/heath it fails
i thought it was to do with tcp listen addr but even when i set that to 0.0.0.0 i still get nothing
vault.hcl

# Ansible managed

cluster_name = "home-lab-autounseal"
max_lease_ttl = "768h"
default_lease_ttl = "768h"

disable_clustering = "False"
cluster_addr = "http://192.168.0.100:8201"
api_addr = "http://192.168.0.100:8200"

plugin_directory = "/usr/local/lib/vault/plugins"

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "192.168.0.100:8200"
  tls_disable = "true"
  }

storage "raft" {
  path = "/opt/vault/data"
  node_id = "vault"
                      retry_join {
    leader_api_addr =  "http://192.168.0.101:8200"
  }
            retry_join {
    leader_api_addr =  "http://192.168.0.102:8200"
  }
        }

// HashiCorp recommends disabling mlock when using Raft.
disable_mlock = true

ui = true

keepalived.conf

global_defs {
    enable_script_security
}
vrrp_script check_vault {
  script: curl http://192.168.0.100/v1/sys/health -f
  interval 5
}

vrrp_instance LB_VIP {
    interface eth0
    state MASTER
    priority 103
    virtual_router_id 100 
    advert_int 1
    virtual_ipaddress {
        192.168.0.200
    }
    track_script {
      check_vault
    }
}

journald for keepalived

Jun 25 08:14:06 vault1-test Keepalived[23927]: Starting Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Jun 25 08:14:06 vault1-test Keepalived[23927]: Running on Linux 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) (built for Linux 5.14.0)
Jun 25 08:14:06 vault1-test Keepalived[23927]: Command line: '/usr/sbin/keepalived' '--dont-fork' '-D'
Jun 25 08:14:06 vault1-test Keepalived[23927]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 25 08:14:06 vault1-test Keepalived[23927]: Configuration file /etc/keepalived/keepalived.conf
Jun 25 08:14:06 vault1-test Keepalived[23927]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jun 25 08:14:06 vault1-test Keepalived[23927]: Starting VRRP child process, pid=23928
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: Registering Kernel netlink reflector
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: Registering Kernel netlink command channel
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: (/etc/keepalived/keepalived.conf: Line 5) Unknown keyword 'script:'
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: (/etc/keepalived/keepalived.conf: Line 7) No script set for vrrp_script check_vault - removing
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: (/etc/keepalived/keepalived.conf: Line 19) (LB_VIP) track script check_vault not found, ignoring...
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: Assigned address 192.168.0.100 for interface eth0
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: Assigned address fe80::be24:11ff:fed7:eff1 for interface eth0
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: Registering gratuitous ARP shared channel
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: (LB_VIP) removing VIPs.
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: (LB_VIP) Entering BACKUP STATE (init)
Jun 25 08:14:06 vault1-test Keepalived_vrrp[23928]: VRRP sockpool: [ifindex(  2), family(IPv4), proto(112), fd(12,13) multicast, address(224.0.0.18)]
Jun 25 08:14:06 vault1-test Keepalived[23927]: Startup complete
Jun 25 08:14:06 vault1-test systemd[1]: Started LVS and VRRP High Availability Monitor.

journald for vault

Jun 25 08:36:47 vault1-test sh[34239]: ==> Vault server configuration:
Jun 25 08:36:47 vault1-test sh[34239]:              Api Address: http://192.168.0.100:8200
Jun 25 08:36:47 vault1-test sh[34239]:                      Cgo: disabled
Jun 25 08:36:47 vault1-test sh[34239]:          Cluster Address: https://192.168.0.100:8201
Jun 25 08:36:47 vault1-test sh[34239]:               Go Version: go1.14.7
Jun 25 08:36:47 vault1-test sh[34239]:               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "192.168.0.100:8200", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Jun 25 08:36:47 vault1-test sh[34239]:                Log Level: info
Jun 25 08:36:47 vault1-test sh[34239]:                    Mlock: supported: true, enabled: false
Jun 25 08:36:47 vault1-test sh[34239]:            Recovery Mode: false
Jun 25 08:36:47 vault1-test sh[34239]:                  Storage: raft (HA available)
Jun 25 08:36:47 vault1-test sh[34239]:                  Version: Vault v1.5.5
Jun 25 08:36:47 vault1-test sh[34239]:              Version Sha: f5d1ddb3750e7c28e25036e1ef26a4c02379fc01
Jun 25 08:36:47 vault1-test sh[34239]: ==> Vault server started! Log data will stream in below:
Jun 25 08:36:47 vault1-test sh[34239]: 2024-06-25T08:36:46.997Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
Jun 25 08:36:47 vault1-test sh[34239]: 2024-06-25T08:36:47.003Z [INFO]  core: raft retry join initiated

curl outputs
``
curl http://192.168.0.100:8200/v1/sys/health ( real ip)
{“initialized”:true,“sealed”:false,“standby”:false,“performance_standby”:false,“replication_performance_mode”:“disabled”,“replication_dr_mode”:“disabled”,“server_time_utc”:1719304347,“version”:“1.5.5”,“cluster_name”:“home-lab-autounseal”,“cluster_id”:“c602f0c9-504a-abb4-9312-51ef869ec1d9”}

curl http://192.168.0.200:8200/v1/sys/health (virtual ip provided by keepalived)
curl: (7) Failed to connect to 192.168.0.200 port 8200 after 5 ms: Couldn’t connect to server
ping 192.168.0.200
PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data.
64 bytes from 192.168.0.200: icmp_seq=1 ttl=64 time=3.71 ms
64 bytes from 192.168.0.200: icmp_seq=2 ttl=64 time=3.44 ms
64 bytes from 192.168.0.200: icmp_seq=3 ttl=64 time=6.41 ms
``

2

I think maybe cluster_address in your TCP stanza may be wrong. Here is an example config I use when I need to test:

ui            = true
cluster_addr  = "http://127.0.0.1:8201"
api_addr      = "https://127.0.0.1:8200"
disable_mlock = true

storage "raft" {
  path    = "./vault-data"
  node_id = "ip-foo.bar"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable = 1
}

(Similar question here for reference: Having trouble bringing up Vault node w/ integrated storage)

3

if i try to change cluster address it fails to join the cluster and genrally all ha stuff fails which is ecpected as i was changing the cluster address to 0.0.0.0 but wheni change the tcp listener to 0.0.0.0 and keep the cluster address to the acctual ip of the vm
log snippet of 2nd node

core: attempting to join possible raft leader node: leader_addr=http://192.168.0.100:8200
Jun 27 18:58:09 vault2-test sh[3763]: 2024-06-27T18:58:09.509Z [INFO]  core: join attempt failed: error="could not retrieve raft bootstrap package"

if i change the tcp listen address back to its actual ip (not 0.0.0.0) it works again
(on further inspection the role im using the ability toadd add extra tcp servers is broken so even setting up a tcp listener to it own ip address like it does usually doesn’t work)