How can I have a robust vault agent configuration, using AppRole

Vault
1

Hello,

Currently we are using vault agent in multiple scenarios - for example, a Jenkins machine using it in order to create a vault token to be used in builds.

The configuration was done like this:

  • vault-agent service in systemd
  • single hcl file, with a configuration similar to the below
pid_file = "./pidfile"

vault {
        address = "https://vault.mycompany.com"
}

auto_auth {
        method "approle" {
                mount_path = "auth/approle"
                config     = {
                    role_id_file_path                   = "/tmp/vault-role-id"
                    secret_id_file_path                 = "/tmp/vault-secret-id"
                    remove_secret_id_file_after_reading = "false"
                }
        }

        sink "file" {
                config = {
                    path = "/opt/vault-token/.vault-token"
                    mode = 0644
                }
        }

}

(NOTE: when deploying the secret ID file, we remove it with the configuration management tool instead of using the option)

The approles themselves are very simple, we create them like {"policies": "nginx"} where the nginx policy is something like this:

  - name: nginx
    policies:
      path:
        - "kv/project/*":
            capabilities:
              - read
              - list
        - "kv/abzu/keycloak/apps-secrets":
            capabilities:
              - read
              - list

This works perfectly, as long as nothing breaks (e.g. VM is unexpectedly restarted). In those cases, even though the vault token in the sink is valid, the agent will ignore it and try to reauthenticate with the role/secret ID, therefore causing the refresh loop to break: any apps depending on the token being valid will continue to work until it expires, and we will notice something is wrong when it expires.

What are our options here to avoid this? Persisting the secret ID on the machine permanently? Building a middleware that gets a new secret ID regularly?

Thanks.