All nomad job switched to pending state after vault outage

I try to investigate why all nomad jobs switched to pending state after vault outage. That was running jobs before vault outage.
Nomad client produced the logs:

 (retry attempt 8 after "32s")
    2021/08/07 18:22:26.710681 [WARN] (view) Error making API request.

Code: 503. Raw Message:

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.

How can I keep running jobs ‘running’ as long as I can after vault outage?

Hi @mbrezovsky :wave:

Are you using a template in your job that reads data from Vault?

@lgfa29 yes, I am using template stanza for vault secrets.

any ideas how to solve that?

There isn’t a way to control retries for templates right now. Could you open a feature request for this in our repo?