Logrotate vault log and audit files - re starting vault

Vault
1

Hello all,

After configuring logrotation for Vault log and audit files, Vault stopped writing to the respective files.

The documentation tells to send a SIGHUP after each rotation, which I tried but Vault service remained ‘dead’. The config reload succeeds though. Is there any extra configuration I shoud be aware of?

Please find below my actual configuration:

  1. /usr/lib/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault_main.hcl
StartLimitInterval=60

[Service]
User=vault
Group=bin
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/bin/sh -c 'VAULT_ADDR=${VAULT_ADDR} VAULT_TOKEN=${VAULT_TOKEN} /usr/local/bin/vault server -config=/etc/vault.d/vault_main.hcl -log-level=trace >> /var/log/vault/vault.log 2>&1'
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=524288
LimitNPROC=524288
LimitMEMLOCK=infinity
EnvironmentFile=/home/vault/.creds
[Install]
WantedBy=multi-user.target
  1. /etc/logrotate.d/vault
/var/log/vault/*.log {

    missingok
    ifempty
    compress # log files will be compressed
    dateext
    dateformat -%Y%m%d-%s
    rotate 20
    size 100M
    maxsize 100M
    create 0664 vault bin
    postrotate
      /usr/bin/systemctl reload vault 2> /dev/null || true
    endscript

}

/var/log/audit/vault_audit.log {


    missingok
    ifempty
    compress # log files will be compressed
    dateext
    dateformat -%Y%m%d-%s
    rotate 20
    size 100M
    maxsize 100M
    create 0664 vault bin
    postrotate
      /usr/bin/systemctl reload vault 2> /dev/null || true
    endscript

}

Thank you for your help!

2

I forgot to specify that I am using the raft integrated storage v1.5.0.

Another misunderstood point for me is that reloading and starting Vault daemon results in the following error

# systemctl reload vault.service
# systemctl start vault.service 
# grep -ri timeout /var/log/vault/vault.log 
Error initializing storage of type raft: failed to create fsm: failed to open bolt file: timeout

By completely stopping Vault and restarting it, it manages to start:

# systemctl stop vault.service && pkill -u vault && systemctl start vault.service

it can be reproduced by using

# systemctl restart vault.service

As there are two processes, one is a child for the other, running the vault server:

# ps -ef | grep vault
vault     1537     1  0 16:42 ?        00:00:00 /bin/sh -c VAULT_ADDR=https://x.x.x.x:8200  /usr/local/bin/vault server -config=/etc/vault.d/vault_main.hcl -log-level=trace >> /var/log/vault/vault.log 2>&1
vault     1538  1537 14 16:42 ?        00:00:00 /usr/local/bin/vault server -config=/etc/vault.d/vault_main.hcl -log-level=trace
# systemctl stop vault
# ps -ef | grep vault
vault     1538     1  4 16:42 ?        00:00:06 /usr/local/bin/vault server -config=/etc/vault.d/vault_main.hcl -log-level=trace

the “systemctl stop vault” command just kills the parent process as you see. that’s why I added the “pkill -u vault” and that’s why Vault timeouts trying to access its db when executing “systemctl restart vault” for example (or the reload/start), as the child process is still locking the db.

 # lsof /var/vault/storage/
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
vault   1538 vault  mem-W  REG  253,1 25169920 67170369 /var/vault/storage/raft/raft.db
vault   1538 vault  mem-W  REG  253,1  2097152       67 /var/vault/storage/vault.db
vault   1538 vault    7uW  REG  253,1  2097152       67 /var/vault/storage/vault.db
vault   1538 vault    8uW  REG  253,1 25169920 67170369 /var/vault/storage/raft/raft.db

# lslocks  | grep vault
vault            1538  FLOCK   2M WRITE 0     0   0 /var/vault/storage/vault.db
vault            1538  FLOCK  24M WRITE 0     0   0 /var/vault/storage/raft/raft.db

Is there any recommandations regarding this?

Thank you for your help again!

3

Changing KillMode=process to KillMode=control-group allowed me to get the desired behaviour while stopping and restarting vault.service.

However, still not sure how Vault handles a SIGHUP, as it’s not restarted when receiving one.