Consul-teamplate not able to renew vault token K8s

Vault
1

Guys, I have installed Vault on GKE cluster and I have deployed my API services to the different GKE clusterts. All the API uses some env & secrets from vault server using kubernetes authentication. All working fine. But after 1 month application couldn’t able to fetch secret and consul-template continuously restarting and lagging my applications. My vault secret default lease configure default_lease_ttl 768h. So as per my finding application break the connection after completed given lease ttl. Is there any way to increase these default TTL or set it to maximum ?

I am getting below error.

2020-04-14T10:38:02.245933588Z URL: PUT https://my-vault:8200/v1/auth/token/renew-self
 E 
2020-04-14T10:38:02.245940057Z Code: 403. Errors:
 E 
2020-04-14T10:38:02.245946053Z 
 E 
2020-04-14T10:38:02.245951689Z * permission denied
 E 
2020-04-14T10:38:02.245957718Z 2020/04/14 10:38:02.245743 [WARN] vault.token: renewer done (maybe the lease expired)
 E 
2020-04-14T10:38:02.245992681Z 2020/04/14 10:38:02.245822 [WARN] (view) lease expired or is not renewable (retry attempt 6 after "32s")
 E 
2020-04-14T10:38:34.213575145Z 2020/04/14 10:38:34.213418 [WARN] (view) vault.read(secrets/kubernetes/cluster/stage/myservice/config): vault.read(secrets/kubernetes/cluster/stage/myservice/config): Error making API request.
 E 
2020-04-14T10:38:34.213623165Z 
 E 
2020-04-14T10:38:34.213631105Z URL: GET https://my-vault:8200/v1/secrets/kubernetes/cluster/stage/myservice/config
 E 
2020-04-14T10:38:34.213637531Z Code: 403. Errors:
 E 
2020-04-14T10:38:34.213657305Z 
 E

here is my Consul template :

vault {
  vault_agent_token_file = "/home/vault/.vault-token"
  renew_token = true
  ssl {
    ca_cert = "/etc/vault/tls/ca.pem"
  }
  retry {
    backoff = "1s"
  }
}

template {
      contents = <<EOF
      {{with secret "secrets/kubernetes/cluster/stage/myservice/config"}}
      {{.Data.env}}
      {{end}}
    EOF
    destination = "/etc/secrets/.env"
   }
2

Hi,

Not sure if you still need an answer to this, but here goes:

As far as I know, consul-template doesn’t rotate tokens. My suggestion would be to run “vault agent” as a sidecar container, and let this do the Kubernetes authentication and token rotation.

Some resources:

To summarize: “Vault agent” will authenticate with Vault using Kubernetes authentication, and write the Vault token to a file (called a “sink”).

“Vault agent” will re-authenticate with Vault if the token expires and generate a new one. Consul-template will re-read the file defined in vault_agent_token_file automatically every x minutes.

1 Like