Wal logs not immediately sent to the secondary DR


my configuration
[Vault 1.13.2+ent.hsm]

Vault e 5 node cluster
Vvault 5 node cluster

I noticed that some operations, such as the disable of a secret engine, are not carried immediately towards the secondary cluster. The last WAL entry on the primary remains ahead of about 5 units compared to the last_remote_wal on the secondary.
This is in an idle situation, where the cluster is doing nothing.
A simple “write” operation is enough for the log to line up
Is this the expected behavior?
Is it possible to force the replication of all operations?

As this is an Enterprise-only feature, definitely open a support case at support.hashicorp.com if you haven’t already.

Should you get a useful answer, and are willing and able to post it here, please do indulge our curiosity!

Here is the answer from Official Support:
The behavior you observed is expected in Vault replication. Replication in Vault is asynchronous by design, so it is possible for the WAL on the primary cluster to be ahead of the WAL on the secondary cluster, even when the clusters are idle. When the primary cluster performs a write operation, it generates a Write-Ahead-Log (WAL) entry. The primary cluster sends the WAL entry to the secondary cluster through log shipping. The secondary cluster applies the WAL entry and updates its merkle index accordingly.

However, Vault does not replicate every operation immediately to the secondary cluster, as it batches WAL entries to optimize performance. Therefore, it is possible that some operations may not be immediately replicated to the secondary cluster, leading to a lag between the last WAL entry on the primary and the last_remote_wal on the secondary.