Hi all,
I’m hoping someone can help me understand a startup panic and, more importantly, the right way to prevent it.
Setup
- Vault with Integrated Storage (Raft), running in Linux containers.
- I drive Vault through a custom automation wrapper (it handles init/unseal/config rendering and lifecycle).
- Two sites, active/standby. Replication between them is done by my wrapper: it periodically pulls a snapshot from the active site and applies it to the standby using a forced snapshot restore (
sys/storage/raft/snapshot-force). This runs on a schedule, so the standby is restored many times over its lifetime. - Over months of running, the standby’s storage files grew to:
raft.db(raft log store): ~4.4 GBvault.db(FSM): ~17 MB
What happened
I recently upgraded both sites to Vault 1.20.2. After the upgrade, the standby node will not start — Vault panics during storage setup while opening the raft log store. The FSM (vault.db) opens fine in ~2 ms; the panic is on the large raft.db:
```text [DEBUG] storage.raft.fsm: time to open database: elapsed=2.29655ms path=/opt/vault/data/vault.db panic: runtime error: index out of range [4649316352] with length 2147483647 goroutine 1
[running]: …/go.etcd.io/bbolt@v1.4.0-beta.0/db.go:1114
…/bbolt@v1.4.0-beta.0/tx.go:594
…/bbolt@v1.4.0-beta.0/tx.go:607
…/bbolt@v1.4.0-beta.0/tx_check.go:130
…/bbolt@v1.4.0-beta.0/db.go:1251
…/bbolt@v1.4.0-beta.0/db.go:422
…/bbolt@v1.4.0-beta.0/db.go:299
…/raft-boltdb/v2@v2.3.0/bolt_store.go:79
…/vault/physical/raft/raft.go:564 main.main() ```
The vault was working fine with the previous version even with 4.4 GB raft.db. Is this a bug?