Vault dynamic creds renewal time wrong, causing apps to crash

In some of our apps when the vault daemon tries to renew database creds the renewal time is invalid (should be 3600s). When this happens, the app crashes. If the renewal time is 3600s, the app continues to function.

time=“2022-06-27T11:40:07Z” level=info msg=“secret database/creds/db-redacted renewed for 1656s” app=vault-env
time=“2022-06-27T11:40:07Z” level=info msg=“secret renewal for database/creds/db-redacted has stopped, sending SIGTERM to process” app=vault-env error=“”
time=“2022-06-27T11:40:07Z” level=info msg=“received signal: terminated” app=vault-env

Paste the current configuration of that database connection as well as the role you’re using – there isn’t anything here we can use to help.

Sorry aram,

Here’s the config:

            - name: redacted
              plugin_name: "mysql-aurora-database-plugin"
              connection_url: "{{username}}:{{password}}@tcp(redacted-db.redacted:3306)/"
              allowed_roles: [db-redacted]
              username: "vault"
              password: "${ awskms (env `REDACTED_VAULT_DB_PASSWORD`) }" # This should be added in the Vault Object
              rotate: false

And the app are uses the following as envs:

            - name: JDBC_DB_USER
              value: 'vault:database/creds/db-redacted#username'
            - name: JDBC_DB_PASSWORD
              value: 'vault:database/creds/db-redacted#password'

Also here’s my Vault config (I removed the vault db connection string from it):

apiVersion: vault.banzaicloud.com/v1alpha1
  kind: Vault
  metadata:
    labels:
    name: vault
    namespace: vault
  spec:
    annotations:
      common/annotation: "true"
      iam.amazonaws.com/role: redacted
    bankVaultsImage: ghcr.io/banzaicloud/bank-vaults:1.14.4
    bankVaultsVolumeMounts:
    - mountPath: /rds-tls
      name: rds-ca-bundle
    caNamespaces:
    - '*'
    config:
      api_addr: https://vault.vault:8200
      listener:
        tcp:
          address: 0.0.0.0:8200
          telemetry:
            unauthenticated_metrics_access: true
          tls_cert_file: /vault/tls/server.crt
          tls_key_file: /vault/tls/server.key
      storage:
        mysql:
          address: redacted
          ha_enabled: "true"
          max_parallel: "30"
          password: ${ awskms (env `MYSQL_BACKEND_PASSWORD`) }
          tls_ca_file: /rds-tls/rds-ca-bundle.pem
          username: vault
      telemetry:
        disable_hostname: true
        prometheus_retention_time: 1h
      ui: true
    credentialsConfig:
      env: ""
      path: ""
      secretName: ""
    envsConfig:
    - name: VAULT_REDACTED_PASSWORD
      value: redacted
    - name: AWS_REGION
      value: us-east-1
    - name: AWS_ACCOUNT_ID
      value: "redacted"
    - name: ENVIRONMENT
      value: prod
    existingTlsSecretName: vault-tls
    externalConfig:
      audit:
      - description: File based audit logging device
        options:
          file_path: /vault/logs/vault.log
          mode: "0640"
        type: file
      auth:
      - config:
          default_role: redacted
          oidc_client_id: vault
          oidc_client_secret: ${ awskms (env `VAULT_CLIENT_SECRET`) }
          oidc_discovery_ca_pem: redacted
          oidc_discovery_url: https://dex.${ env `ENVIRONMENT` }.local
        roles:
        - allowed_redirect_uris: https://vault.${ env `ENVIRONMENT` }.local/ui/vault/auth/oidc/oidc/callback
          bound_claims:
            groups:
            - redacted
          name: redacted
          oidc_scopes: profile,openid,email,groups
          policies: admin
          role_type: oidc
          ttl: 1h
          user_claim: email
        - allowed_redirect_uris: https://vault.${ env `ENVIRONMENT` }.local/ui/vault/auth/oidc/oidc/callback
        type: oidc
      policies:
      - name: admin
        rules: |
          path "sys/auth" {
            capabilities = ["read"]
          }
          path "auth/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "sys/auth/*" {
            capabilities = ["create", "read", "update", "delete", "sudo"]
          }
          path "identity/*" {
            capabilities = [ "create", "read", "update", "delete", "list" ]
          }
          path "sys/policies/acl" {
            capabilities = ["read","list"]
          }
          path "sys/policies/acl/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "secrets/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "aws/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "database/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "pki/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "rabbitmq/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "sys/mounts" {
            capabilities = ["read"]
          }
          path "sys/mounts/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
          path "sys/health" {
            capabilities = ["read", "sudo"]
          }
          path "sys/metrics" {
            capabilities = ["read"]
          }
          path "sys/capabilities" {
            capabilities = ["create", "update"]
          }
          path "sys/capabilities-self" {
            capabilities = ["create", "update"]
          }
          path "sys/leases/*" {
            capabilities = ["create", "read", "update", "delete", "list", "sudo"]
          }
      secrets:
      - description: General secrets for environment.
        options:
          version: 2
        path: secrets
        type: kv
    fluentdConfFile: fluent.conf
    fluentdConfig: |
      <source>
        @type tail
        format json
        keep_time_key true
        time_format %iso8601
        path /vault/logs/vault.log
        pos_file /vault/logs/vault.log.pos
        tag s3.vault.audit
      </source>
      <match s3.*.*>
        @type s3
        s3_bucket redacted
        s3_region us-east-1
        path audit-logs/
        time_slice_format %Y%m%d%H
        time_slice_wait 10m
        utc
      </match>
    fluentdEnabled: true
    fluentdImage: banzaicloud/fluentd-s3:latest
    image: vault:1.10.4
    nodeAffinity: {}
    nodeSelector:
      node-role.kubernetes.io/master: ""
    podAntiAffinity: failure-domain.beta.kubernetes.io/zone
    resources:
      bankVaults:
        limits:
          cpu: 500m
          memory: 128Mi
        requests:
          cpu: 100m
          memory: 64Mi
      prometheusExporter:
        limits:
          cpu: 500m
          memory: 128Mi
        requests:
          cpu: 100m
          memory: 64Mi
      vault:
        limits:
          cpu: "2"
          memory: 2Gi
        requests:
          cpu: 200m
          memory: 256Mi
    securityContext: {}
    serviceAccount: vault
    serviceMonitorEnabled: true
    size: 3
    statsdDisabled: true
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
      operator: Exists
    unsealConfig:
      aws:
        kmsKeyId: redacted
        kmsRegion: us-east-1
        s3Bucket: redacted
        s3Prefix: redacted
        s3Region: us-east-1
      kubernetes: {}
      options:
        preFlightChecks: true
    vaultAnnotations:
      type/instance: vault
    vaultConfigurerPodSpec: {}
    vaultContainerSpec:
      name: ""
      resources: {}
    vaultEnvsConfig:
    - name: MYSQL_BACKEND_PASSWORD
      value: redacted
    - name: AWS_REGION
      value: us-east-1
    vaultPodSpec: {}
    volumeMounts:
    - mountPath: /rds-tls
      name: rds-ca-bundle
    volumes:
    - configMap:
        name: rds-ca-bundle
      name: rds-ca-bundle

Let me know if anything else is needed

I’m not sure if I understand this – maybe read the config/role from Vault itself? Also need to pull the system log from Vault to get the actual errors (if there is no information you may need to set the vault log level to debug to capture it).

Hi @chrism417 ,

Welcome!

Some of the terminology you’re using here is not correct:

A lease being renewed for less than 3600s is not “invalid” - it simply means configured limits have restricted the renewal time.

The app is not “crashing”. It is being deliberately and intentionally stopped once a secret it is using is no longer renewable.

All Vault leases have TTLs, which are configured in a hierarchy:

  • There is a system-wide default-maximum TTL for the Vault process, which is 32 days unless overridden in the Vault server config file.

  • There is a maximum TTL configurable via mount tuning, for each secret engine and auth method. (So in this case, you’d care about the one for the database secret engine you are using.)

  • There is a maximum TTL defined as part of each role configured within the database secrets engine.

Once the applicable maximum is reached, Vault will decline to extend the lease further, by design, requiring new credentials are fetched.