Vault returns "HTTP Error 500" when generating 1000 RSA keys

Vault
1

Hello, everyone!
I’d like to generate 100k RSA-3072 certificates with PKI engine, but Vault starts to respond with HTTP 500 error after ~800th request:

“error occurred: unable to store certificate locally: context canceled”

The source of the error is probably here

It seems that Vault is unable to keep up with high rate of simultaneous certificate generation requests.

So I’d like to know the following:

  1. is there any way or best practice to avoid this HTTP 500 error?
  2. is there some Vault setting it’s worth to look at?

I published a binary reproducing the issue, you can run it like this:
./vault-cert-gen --keys-count 1000
Here’s the typical run on my machine:

Generated certificate, key type: rsa
...
// 948 successful certificate generations
...
Generate certificate error: The Vault server returned an error (status code 500)
...
// 52 similar errors
...
generating 1000 keys, errors: 52
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `52`,
 right: `0`', src/main.rs:54:9

Which means there were 52 errors out of 1000 generated certificates (the number of errors is different on each run, can be 930 errors, 4 errors, etc.)

My setup: Docker Vault from here/12 core AMD Ryzen 5 pro 5650u/32GB RAM

Any suggestions will be greatly appreciated
Thank you!

2

From my experience with the PKI secrets engine, it is not well suited to generating that many certificates if you expect them to be kept in storage. It inflates the total size of the overall Vault storage hugely, and results in tidy operations then taking a long time.

I’m not that familiar with Rust, but is that telling it to make 1000 requests to Vault broadly simultaneously and then wait for them all to complete?

I’m not hugely surprised that some would encounter difficulty, at that level of concurrency.

3

thank you for the reply

1000 requests to Vault broadly simultaneously and then wait for them all to complete?

AFAIK it’s up to the runtime and it probably indeed spins up 1k tasks that are being processed by thread pool