Vault GCS backend - Class A Operations usage

Vault
1

Hello,

I am running a successful Vault deployment in HA configuration, based on Google Cloud Storage backend. The config is as follows:

ui = true
log_level = "info"
listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}
service_registration "kubernetes" {}
storage "gcs" {
  bucket     = "REDACTED"
  ha_enabled = "true"
}
seal "gcpckms" {
   project     = "REDACTED"
   region      = "REDACTED"
   key_ring    = "REDACTED"
   crypto_key  = "REDACTED"
}

This configuration utilizes TONS of GCS API Class A Operations and Class B Operations:

image
image1854×598 64 KB

Google charges for that API calls additionally. The number are pretty high and cost me hundreds of dollars per month. Is there anything we can do to decrease the usage of GCS?

image
image986×358 32.1 KB

Any help appreciated!
Thanks,
Szymon

2

Found the reason of the problem. FYI:

External Secrets Operator creates a lease for every secret sync. With multiple K8s clusters and over 500 secrets it was creating the leases constantly. Setting experimental-enable-vault-token-cache and setting longer TTLs to auth methods helped me to reduce the GCS API calls.

image
image1860×596 63.3 KB

3

You might find it useful to also switch to batch type tokens whereever you can, as well.

They are not as flexible as service tokens, but they require no storage operations to create or validate, which may be particularly relevant in your scenario.

However, be aware that Vault does have various background processes that need to invoke ListObjects periodically - which seems unfortunate given the GCS storage pricing system :-/