All nomad job switched to pending state after vault outage

Nomad
1

I try to investigate why all nomad jobs switched to pending state after vault outage. That was running jobs before vault outage.
Nomad client produced the logs:

 (retry attempt 8 after "32s")
    2021/08/07 18:22:26.710681 [WARN] (view) vault.read(secret/infra/influx): vault.read(secret/infra/influx): Error making API request.

URL: GET http://10.0.0.62:8200/v1/secret/data/infra/influx
Code: 503. Raw Message:

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

How can I keep running jobs ‘running’ as long as I can after vault outage?

2

Hi @mbrezovsky :wave:

Are you using a template in your job that reads data from Vault?

3

@lgfa29 yes, I am using template stanza for vault secrets.

4

any ideas how to solve that?

5

There isn’t a way to control retries for templates right now. Could you open a feature request for this in our repo?

Thanks!