Consul template pod restarting kubernetes

Vault
1

We have deployed our apps on kubernetes GCP. Our secrets are stored in Vault. I have deployed vault-agent as init container and consul template along with my applications. Now everything is working properly. but consul template keep restarting and my application goes down due to that. After restarting my application pod. Consul template keep working till one month. after a month it again restating. So i have to restart my application pod every time when consul template restats.

Error :

E 2020-01-18T18:27:15.075280671Z * permission denied (retry attempt 7 after "1m0s")
 
E 2020-01-18T18:28:15.091879571Z 2020/01/18 18:28:15.091720 [WARN] (view) vault.read(secret/kubernetes/cluster/prod/job/config): vault.read(secret/kubernetes/cluster/prod/job/config): Error making API request.
 
E 2020-01-18T18:28:15.091920823Z 
 
E 2020-01-18T18:28:15.091932805Z URL: GET https://my-vault:8200/v1/secret/kubernetes/cluster/prod/job/config
 
E 2020-01-18T18:28:15.091939006Z Code: 403. Errors:

Below are my config files.

vault {
  vault_agent_token_file = "/home/vault/.vault-token"
  ssl {
    ca_cert = "/etc/vault/tls/ca.pem"
  }
  retry {
    backoff = "1s"
  }
}

template {
      contents = <<EOF
      {{with secret "secret/kubernetes/cluster/prod/job/config"}}
      {{.Data.env}}
      {{end}}
    EOF
    destination = "/etc/secrets/.env"
   }

vault-agent-config.hcl

# Uncomment this to have Agent run once (e.g. when running as an initContainer)
exit_after_auth = true
pid_file = "/home/vault/pidfile"



auto_auth {
    method "kubernetes" {
        mount_path = "auth/prod-cluster"
        config = {
            role = "prod"
        }
    }

    sink "file" {
        config = {
            path = "/home/vault/.vault-token"
        }
    }
}

listener "tcp" {
   address = "127.0.0.1:8200"
   tls_cert_file = "/etc/vault/tls/vault.pem"
   tls_key_file = "/etc/vault/tls/vault-key.pem"
   tls_min_version = "tls12"

}

vault {
   address = "https://my-vault:8200"
}

My vault role :

**{"request_id":"470c5017-6aaa-19b0-c58a-e6f11f4f34a2","lease_id":"","renewable":false,"lease_duration":0,"data":{"bound_cidrs":[],"bound_service_account_names":["vault-auth"],"bound_service_account_namespaces":["default"],"max_ttl":0,"num_uses":0,"period":0,"policies":["my-policy"],"ttl":86400},"wrap_info":null,"warnings":null,"auth":null}
**

Let me know if you want more info.

2

Hi @imvishalvyas,

Sounds like the vault token that you are retrieving via Agent is expiring after 1 month. You may want to increase the TTL configured on the kubernetes auth method or, better yet, switch to using a periodic token which stays active so long as a process is continually renewing it. See these docs for more information: https://www.vaultproject.io/docs/concepts/tokens/#periodic-tokens

Hope this helps!

3

I also just noticed in the documentation for vault_agent_token_file

by default Consul Template will not try to renew the Vault token, if you want it
to renew you will need to specify renew_token = true as below.

So try adding renew_token = true to the vault section of the consul template config

4

Yes @briankassouf , You right. My pod get restarts after 1 month. I have checked that and got to know that I have authenticated my kubernetes cluster using below command.

vault write auth/prod-cluster/role/prod bound_service_account_names=vault-auth bound_service_account_namespaces=default policies=prod-policy ttl=24h

I didn’t understand why it was expires after 1 month while i have set it to the 24h ttl time.

Also my requirement is to run my application long time and token should not expire till i restart the pod manually. As you suggested that use periodic token, i checked the documentation. But I didn’t understand that how to apply it in my configuration. Is there any detailed document of it ?

And the last i have added renew_token = true in my consul template config file, adding this token will never expire, right ?