Vault agent doesn't respect the number of retries

Vault
1

Hi everyone,
I’d like to limit the number of retries for the Vault agent. The config I use:

{
  "auto_auth": {
    "method": {
      "type": "kubernetes",
      "mount_path": "auth/kube_cluster_c5",
      "config": {
        "role": "secrets-test-dev"
      }
    },
    "sink": [
      {
        "type": "file",
        "config": {
          "path": "/vault/vault-token"
        }
      }
    ]
  },
  "cache": {
    "use_auto_auth_token": "true"
  },
  "exit_after_auth": false,
  "pid_file": "/vault/vault.pid",
  "vault": {
    "address": "https://vault.service:8200",
    "retry": [
      {
        "num_retries": "3"
      }
    ]
  },
  "template_config": [
    {
      "error_on_missing_key": false,
      "exit_on_retry_failure": false
    }
  ],
  "template": [
    {
      "destination": "/secrets/secret-new",
      "contents": "{{ with secret \"/dev/kv/secrets/app1/secret-new\" }}{{ .Data.data.value }}{{ end }}",
      "left_delimiter": "{{",
      "right_delimiter": "}}"
    }
  ],
  "listener": [
    {
      "type": "tcp",
      "address": "127.0.0.1:8200",
      "tls_disable": true
    }
  ]
}

According to the docs, I expect that the Vault agent won’t exit on an error and that the number of retries will be three. This is what I’m getting:

2022-05-16T14:32:48.342Z [DEBUG] cache.apiproxy.client: performing request: method=GET url=https://vault.service.confiad1.consul.csnzoo.com:8200/v1/dev/kv/secrets/app1/secret-new
2022-05-16T14:32:48.346Z [WARN] (view) vault.read(dbtech/dev/db/spdb/static-creds/svc-nrailic-secrets-test2-app): no secret exists at dbtech/dev/db/spdb/static-creds/svc-nrailic-secrets-test2-app (retry attempt 7 after "16s")
2022-05-16T14:32:48.347Z [WARN] (view) vault.read(dev/kv/secrets/app1/secret-new): no secret exists at dev/kv/secrets/app1/secret-new (retry attempt 7 after "16s")

I tried to set the VAULT_MAX_RETRIES ENV variable but no luck. Also, tried to set it to -1 but this also doesn’t work:

/vault $ export VAULT_MAX_RETRIES=-1
/vault $ ./vault agent --tls-skip-verify --config=/vault/config.json --log-level debug
Error fetching client: failed to read environment: strconv.ParseUint: parsing "-1": invalid syntax

I can tell that the config is being parsed properly, because if I pass two values in the retry stanza I got an error that only one is allowed.

There is an option to pass the Consul template retry config but the problem is that I don’t understand how to pass it from the Vault config.

This is Consul template config for the example above:

[DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":33,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"{{ with secret \"/dev/kv/secrets/app1/secret-new\" }}{{ .Data.data.value }}{{ end }}","CreateDestDirs":true,"Destination":"/secrets/secret-new","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":{},"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":33,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000}

Any help would be appreciated.