we have started using vault for storing private data such as phone numbers, account numbers etc… With this we plan to put quite heavy load on vault (in the direction of 1000 secrets per second at peak time) and we will add several million entries per day.
I realized now, that most people seem to use vault for for storing keys, that they use to encrypt critical data somewhere else. This way of course you don’t have such a heavy load on vault as you might use such a secret maybe for a day or so. The load for storing the actualy (encrypted values) to a different DB.
My question: Is the way we have started to use vault a dead end? Shouldn’t we use vault to store a massive load of encrypted values?
I realized now, that most people seem to use vault for for storing keys, that they use to encrypt critical data somewhere else.
For the scenario that you’ve described, you might want to consider moving towards this model via the Transit Secrets Engine, especially if you are able to re-use keys to encrypt data. This means that the amount of data that Vault needs to store doesn’t increase linearly with the actual amount of plaintext data. You can also leverage Transit’s batch_input functionality to batch encrypt/decrypt data, drastically reducing the number of requests you’ll make against Vault.
Regarding ‘you might consider’:
What are the downsides of using vault/consul in my scenarios? I mean: does Consul - used as backend for vault - have any scalability issues/limits?