# Why is vault using so much memory

**URL:** <https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863>\
**Category:** Vault\
**Tags:** k8s, raft\
**Created:** [November 14, 2022, 3:26pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863 "2022-11-14T15:26:14Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 14, 2022, 3:26pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/1 "2022-11-14T15:26:14Z")

</div>

I’m using 3 vault instances with spinning disks,

I have 5 clusters with a bunch of microservices retrieving secrets from vault using banzaicloud’s vault-secrets-webhook.

I am noticing that vault pods will have a linear increase in memory over time until being killed.

I have problems that vault token lookups take more than 10 seconds at some moment.

I noticed also that because of spinning disks I assume there is a high number of raftCommitTime occassionaly takes 1 second to write which I assume is much to slow.

This is one thing I will try to fix by moving to SSDs instead… ( currently using CEPH for persistent volume ) )

But why Vault is ever increasing in RAM is unclear…

Running vault 1.11.3

It seems like a problem. Does anybody have a suggestion on why this could be happening?

I highly doubt there is a memory leak in vault.

I read something about: [prevent memory leak when using control group factors in a policy by hghaf099 · Pull Request #17532 · hashicorp/vault · GitHub](https://github.com/hashicorp/vault/pull/17532)

but doubt whether i am affected.

Will try to update to 1.11.5 regardless…

---

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 14, 2022, 3:31pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/2 "2022-11-14T15:31:14Z")

</div>

![vaultmemorykill](https://global.discourse-cdn.com/hashicorp/original/3X/2/b/2b5027a4a2b921096069b47b276b4874d03a8415.png)

---

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 14, 2022, 3:33pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/3 "2022-11-14T15:33:26Z")

</div>

![vaultio](https://global.discourse-cdn.com/hashicorp/original/3X/b/3/b32b91f5576fb14dc44ad4f4cf21e54704619494.png)

---

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 15, 2022, 5:10pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/4 "2022-11-15T17:10:09Z")

</div>

we moved to SSDs everywhere to deal with the storage IOPS.

Raft + spinning disks didn’t seem quick enough

---

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 15, 2022, 5:10pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/5 "2022-11-15T17:10:24Z")

</div>

Why there is a linear increase in memory is still unclear though :(\*

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [November 15, 2022, 8:29pm UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/6 "2022-11-15T20:29:25Z")

</div>

It is very difficult to speculate on causes without intimate knowledge of your specific Vault workloads.

Some ideas come to mind though:

- How big is your Raft database on disk? (Could the growth be more and more of the database ending up in RAM over time?)

- HashiCorp have a tool called `hcdiag` for collecting information about a Vault instance. Even if you won’t be engaging HashiCorp commercial support, examining the output yourself might help.

---

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 16, 2022, 10:25am UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/7 "2022-11-16T10:25:26Z")

</div>

Hi @maxb thank you for your reply.

This quite trivial I think.

The raft storage is 160MB

 ![vaultraft](https://global.discourse-cdn.com/hashicorp/original/3X/6/6/66628bcd738c0dc6cf6178515bfcce07c8ed5923.png)

the raftcommittime dropped with a factor ~1000 after moving to SSD.

The Linear increase of ram is quite confusing IMO

We use Banzaicloud’s bank-vault operator and we let it configure vault to use an emptyDir to store the vault\_audit.log

As I understand it, by default an emptyDir volume should be temporarily stored on disk.

However it seems to be part of the memory, which is quite confusing to me.

I truncated the vault\_audit.log with and noticed the memory drop. So it seems the emptyDir is in memory.

 ![vaultmemoryusage](https://global.discourse-cdn.com/hashicorp/original/3X/d/3/d30a8b7fb7798444c62b4ffdde2917fbbebfc606.png)

```auto
  - emptyDir: {}
    name: vault-auditlogs
  - emptyDir:
      medium: Memory
      sizeLimit: 1Mi
    name: vault-config

```

so the vault-config is stored in memory, whilst according to this definition the vault-auditlogs should not.

Quote: " default `emptyDir` volumes are stored on whatever medium that backs the node such as disk, SSD, or network storage, depending on your environment. If you set the `emptyDir.medium` field to `"Memory"`, Kubernetes mounts a tmpfs (RAM-backed filesystem) for you instead. While tmpfs is very fast, be aware that unlike disks, tmpfs is cleared on node reboot and any files you write count against your container’s memory limit."

> **[Volumes](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)**
>
> On-disk files in a container are ephemeral, which presents some problems for non-trivial applications when running in containers. One problem is the loss of files when a container crashes. The kubelet restarts the container but with a clean state. A...

---

<div class="post-metadata">

**Author:** ![maxb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/maxb/32/26655_2.png) [@maxb](https://discuss.hashicorp.com/u/maxb)\
**Post date:** [November 16, 2022, 10:33am UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/8 "2022-11-16T10:33:36Z")

</div>

That sounds like you’re not actually using the audit log at all, if you’re just writing it to a temporary directory that gets wiped on pod termination? So you could just turn it off?

---

<div class="post-metadata">

**Author:** ![squaricdot](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/squaricdot/32/30803_2.png) [@squaricdot](https://discuss.hashicorp.com/u/squaricdot)\
**Post date:** [November 16, 2022, 11:49am UTC](https://discuss.hashicorp.com/t/why-is-vault-using-so-much-memory/46863/9 "2022-11-16T11:49:27Z")

</div>

There is fluentd container in the pod that mounts this directory as well and listens to the vault-audit.log file to ship it.

It’s based on: [bank-vaults/vault\_controller.go at dcdd0c5eb81648259e8f034024a84b61c5ecd7ab · banzaicloud/bank-vaults · GitHub](https://github.com/banzaicloud/bank-vaults/blob/dcdd0c5eb81648259e8f034024a84b61c5ecd7ab/operator/pkg/controller/vault/vault_controller.go#L1704)