Vault performance reading secrets from multiple unique paths vs one path multiple values

Vault
1

Is there a performance difference between the 2?

I’m asking because today my vault cluster was averaging around 2770 vault reads a minute due to one of our internal services spinning up more machines.

Telemetry metrics(not vault provided ones) indicated that it took about ~1 second on average for vault to return all secrets for a given service , when it used to take ~150ms prior to the increased read traffic.

Infrastructure metrics showed nothing out of the ordinary, CPU usage <5%, 200Mb used(k8s pod), running on open source version not enterprise.

Will there be any performance benefits storing secrets in a map under one path vs multiple paths?

2

I don’t know for sure but I don’t think it would matter which route you go.

I think what will make a bigger difference is your storage backend type and configuration. If you’re using something like integrated storage (raft) with slower disks that will likely get constrained faster as you scale up than something like a Consul based storage backend (which is primarily in-memory). I believe best practice for integrated storage (raft) or similar would be to have dedicated disks for the data so you can minimize any read/write queuing, especially with larger loads.

I’m assuming Vault isn’t the only thing running on the Kube cluster, in which case looking at the cluster’s performance as a whole might offer some insight (I’m still fairly inexperienced with Kube so I might be making a bad assumption here).

3

the backend storage is dynamo db(was not setup by me). do you have any experience with it or heard what peoples opinion of it?

4

My guess is that you actually constrained your AWS environment while the new instances were launched more than you were stressing Vault. People forget that you’re still on a piece of hardware somewhere, getting the same performance in a shared environment as you would on your own hardware is not going to be equal. It’s one of the things you’re giving up for a lower TCO.

As far as backends for Vault, if testing with the same backend paths will not make a difference. Using multiple “mounts or engines” will use up for resources but again should not add latency to the responses.

Using consul as a backend, is all memory bound.
Using raft (integrated storage) is all disk bound.
From my perspective only - other backends are “nice” to play with but not best-practice and I would strongly advise against it for production use.

Also – open source and enterprise share the same code base, so there is no difference in performance, afaik. Enterprise just enables some extra features.