Leader node won't step down

Vault
1

I’m fairly new to Vault clusters. I have a setup of 3 nodes, all running version 1.15.4 on Ubuntu 22.04.

When I’m trying to step down the leader node it enters in follower state but within ~10 seconds acquires lock and and becomes the leader again, the other follower nodes are not taking over while they are up & running, so far I haven’t found the reason why they are not taking over.

Here is vault.log on the leader node

2024-01-18T15:08:09.137Z [WARN]  core: stepping down from active operation to standby
2024-01-18T15:08:09.138Z [INFO]  core: pre-seal teardown starting
2024-01-18T15:08:09.639Z [INFO]  core: stopping raft active node
2024-01-18T15:08:09.640Z [INFO]  core: pre-seal teardown complete
2024-01-18T15:08:09.654Z [INFO]  storage.raft: entering follower state: follower="Node at 10.1.60.95:8201 [Follower]" leader-address= leader-id=
2024-01-18T15:08:09.654Z [INFO]  storage.raft: aborting pipeline replication: peer="{Voter node02 10.1.61.122:8201}"
2024-01-18T15:08:09.655Z [INFO]  storage.raft: aborting pipeline replication: peer="{Voter node03 10.1.60.208:8201}"
2024-01-18T15:08:09.810Z [INFO]  storage.raft: entering candidate state: node="Node at 10.1.60.95:8201 [Candidate]" term=455
2024-01-18T15:08:09.821Z [INFO]  storage.raft: election won: term=455 tally=2
2024-01-18T15:08:09.821Z [INFO]  storage.raft: entering leader state: leader="Node at 10.1.60.95:8201 [Leader]"
2024-01-18T15:08:09.822Z [INFO]  storage.raft: added peer, starting replication: peer=node02
2024-01-18T15:08:09.822Z [INFO]  storage.raft: added peer, starting replication: peer=node03
2024-01-18T15:08:09.824Z [INFO]  storage.raft: pipelining replication: peer="{Voter node02 10.1.61.122:8201}"
2024-01-18T15:08:09.842Z [INFO]  storage.raft: pipelining replication: peer="{Voter node03 10.1.60.208:8201}"
2024-01-18T15:08:19.656Z [INFO]  core: acquired lock, enabling active operation
2024-01-18T15:08:19.668Z [INFO]  core: post-unseal setup starting
2024-01-18T15:08:19.682Z [INFO]  core: loaded wrapping token key
2024-01-18T15:08:19.682Z [INFO]  core: successfully setup plugin runtime catalog
2024-01-18T15:08:19.682Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2024-01-18T15:08:19.699Z [INFO]  core: successfully mounted: type=system version="v1.15.4+builtin.vault" path=sys/ namespace="ID: root. Path: "
2024-01-18T15:08:19.700Z [INFO]  core: successfully mounted: type=identity version="v1.15.4+builtin.vault" path=identity/ namespace="ID: root. Path: "
2024-01-18T15:08:19.701Z [INFO]  core: successfully mounted: type=pki version="v1.15.4+builtin.vault" path=winter_pki/ namespace="ID: root. Path: "
2024-01-18T15:08:19.701Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.15.4+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2024-01-18T15:08:19.736Z [INFO]  core: successfully mounted: type=token version="v1.15.4+builtin.vault" path=token/ namespace="ID: root. Path: "
2024-01-18T15:08:19.737Z [INFO]  core: successfully mounted: type=approle version="v1.15.4+builtin.vault" path=approle/ namespace="ID: root. Path: "
2024-01-18T15:08:19.745Z [INFO]  core: restoring leases
2024-01-18T15:08:19.763Z [INFO]  core: starting raft active node
2024-01-18T15:08:19.764Z [INFO]  storage.raft: starting autopilot: config="&{false 0 10s 24h0m0s 1000 0 10s false redundancy_zone upgrade_version}" reconcile_interval=0s
2024-01-18T15:08:19.783Z [INFO]  core: usage gauge collection is disabled
2024-01-18T15:08:19.806Z [INFO]  core: post-unseal setup complete

Any ideas what should I try ?

2

This was solved by creating /var/log/vault_audit.log on the follower node

stat /var/log/vault_audit.log
  File: /var/log/vault_audit.log
  Size: 3987      	Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d	Inode: 1745        Links: 1
Access: (0600/-rw-------)  Uid: (  997/   vault)   Gid: (  997/   vault)
Access: 2024-01-24 20:51:14.512045745 +0000
Modify: 2024-01-24 20:51:11.512052443 +0000
Change: 2024-01-24 20:51:11.512052443 +0000
 Birth: 2024-01-24 20:50:31.008143257 +0000

now step down works properly, good!

Question: if this file is needed shouldn’t the vault package create it? I have vault/jammy,now 1.15.4-1 amd64 installed.