Vault Performance Problem

Vault
1

Hi community!

I installed Vault on my IoT device and ran Vault by making the necessary settings. I use Filesystem as the storage backend and I use the KV engine as secret engine. Here is my HCL file:

storage “file” {
path = “/mnt/vault/data/test24”
}

listener “tcp” {
address = “127.0.0.1:8200”
tls_cert_file = “/etc/vault.d/vault.crt”
tls_key_file = “/etc/vault.d/vault.key”
}

disable_mlock = true

When I work with Vault on my Ubuntu machine, I don’t have any performance problems. But when I run Vault on the board, I see that it uses about 70% of the CPU when reading or writing, and that Vault operations are slowing down. I use the Yocto operating system on the board. And I use Variscite’s iMX-DART-6UL as an embedded device. (Variscite DART-6UL : NXP iMX6UL / iMX6ULL / iMX6ULZ System on Module / Computer on Module)

What can I do to fix this problem? Many services are running in my project and Vault consumes more than all of them. What can I do to prevent this and reduce Vault’s CPU consumption?

Thanks.

2

I’d say it is definitely not a Vault performance problem, looks to me more like IO bottleneck of this board :slight_smile: Do you use eMMC or NAND? Why do you need Vault permanently running, what if you’d try to run it periodically?

3

Hi @avoidik

I am working on PKI certificates in my project and I save the certificates in Vault. So Vault always be standing. By the way I use eMMC flash in my board.

4

Hi,

Do you use it as a client (i.e. issuing certificates from a remote server), or as a server (i.e. issuing certificates directly on it, locally)?

5

Hi,

I use as a server.

6

What if you try to run Vault server in dev mode? I’m wondering will it suffer the same performance issue?

7

Hi Viacheslav,

[Unit]
Description=vault server

[Service]
Restart=on-failure
ExecStart=/usr/bin/vault server -dev
ExecStop=/usr/bin/vault operator step-down

[Install]
WantedBy=multi-user.target

I changed my .service file on my device, but this is not working in this way:
Following is my error when i run the dev server mode:

[[0;1;31m●[[0m vault.service - vault server
   Loaded: loaded (/lib/systemd/system/vault.service; enabled; vendor preset: enabled)
   Active: [[0;1;31mfailed[[0m (Result: exit-code) since Wed 2020-02-19 09:36:23 UTC; 7s ago
  Process: 1563 ExecStart=/usr/bin/vault server -dev [[0;1;31m(code=exited, status=1/FAILURE)[[0m
 Main PID: 1563 (code=exited, status=1/FAILURE)

Feb 19 09:36:23 imx6ul-var-dart systemd[1]: vault.service: Service RestartSec=100ms expired, scheduling restart.
Feb 19 09:36:23 imx6ul-var-dart systemd[1]: vault.service: Scheduled restart job, restart counter is at 5.
Feb 19 09:36:23 imx6ul-var-dart systemd[1]: Stopped vault server.
Feb 19 09:36:23 imx6ul-var-dart systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39mvault.service: Start request repeated too quickly.[[0m
Feb 19 09:36:23 imx6ul-var-dart systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39mvault.service: Failed with result 'exit-code'.[[0m
Feb 19 09:36:23 imx6ul-var-dart systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to start vault server.[[0m

My main problem is this: Vault missing client token when start on systemd

Can you support me on this? I’m stuck.

8

My guess is that you are running on a system that does not have AES offloading in hardware. Vault does a lot of encryption/decryption, so if your CPU doesn’t have AES capability all of it must be done in the CPU.