Question - does consul offers http endpoint for kv bulk import? i understand consul cmdline support import from file, but haven’t seen similar feature through http endpoint.
The intention is not to enforce atomicity (but certainly not against it). We are evaluating if there’s efficient way to upload O(10K)~O(100K) kv pairs. It would obviously take too long if we can upload one kv pair at a time…
Thanks for your input.
I used a small shell script to load 10,000 secrets into vault and it took 2 minutes, so not sure if you need anything special. This was just using ‘vault kv put’
thansk for the input Aram.
We assumed no access to the executable, and that’s why we resort to http endpoint.
If http endpoint only supports kv put one at a time, assuming 12ms per entry (using Aram’s 10K/2min, presumably mostly on network time?), it just feels inefficient…
We can try to slice and dice and concurrently upload multiple non-overlapping KV pairs into consul KV, but want to see if consul already offers bulk kv import natively .
The binary uses the API http end point as well so that’s not any different.
BTW, if these are “vault” secrets, you cannot import them through consul by-passing vault. You have to use the Vault API to do any of the importing.
One more note, these were secret spread around different paths, ~80 namespaces. Also, this wasn’t all single KV entries, there were cases where 15-18 KVs were being imported into a single secret – that to me is a single operation/secret.
Appreciate the insight aram!
Somehow related, are you familiar with recursive delete and get behavior in consulKV? Do they also operate one-pair-at-a-time under the hood like put? thanks
Another follow up - given our usage pattern, say about 400K kv pairs, and all of them changes roughly O(1) times a day, would this change rate pose performance concern on consul kv store (for instance, defragmentation/etc., nothing specific, just general question)? would be great if there’s best practice reference to consulKV from load/usage pattern perspective. thanks.
update:
I don’t know Consul well enough to answer that. I know Vault and how it uses the Consul backend as storage but that’s about it.
Consul KV recursive reads and deletes can be performed using a single API call from the client by specifying the recurse query option.
The servers process the recursive GET and DELETE requests as a single operation in the state store.
Awesome, appreciate the confirmation and pointer blake! (i was going to mark the reply as solution too, but looks like, understandably, only single solution mark is allowed for this conversation)