"Can't request Vault token for terminal allocation" when parsing template in poststop task

vadim.cesonis · August 5, 2021, 1:44pm

Hi.

Faced strange problem. Have Nomad job with several tasks. One of the task I’d like to execute after main task stops. So lifecycle is set for “poststop”. But it cannot be executed because of strange error “Vault: server failed to derive vault token: Can’t request Vault token for terminal allocation”. And this happens only when task is marked as poststop. If I change nothing but lifecycle to “prestart” everything works perfectly.
What additional information should I provide to get any help? Am spending too much time on this problem and cannot find what is wrong.

Thanks in advance.

tamas · June 29, 2022, 9:45am

I’m having the same problem. Did you find a solution?

harningt · October 16, 2022, 7:07pm

I’m having the same issue as well - I’m wanting to do some final cleanup + backup and w/o vault available cannot get the keys.

I also realized that even before I had a template that used the value - the post-stop action reported a failure if I had the vault dependency declared at the group level.

… interestingly… with the template declared - it tried to render the secrets despite not getting a vault token and failed there… so looks like the system tried to go through the remaining work-items to create the task, even though it wouldn’t have run because the vault token wouldn’t have been issued.

harningt · October 16, 2022, 7:10pm

Relevant code lines:

github.com

hashicorp/nomad/blob/61a6dbcfcb62cc9d727f21a80f659915b65510c1/nomad/node_endpoint.go#L1648


      
          		return nil
          	}
          	if alloc == nil {
          		setError(fmt.Errorf("Allocation %q does not exist", args.AllocID), false)
          		return nil
          	}
          	if alloc.NodeID != args.NodeID {
          		setError(fmt.Errorf("Allocation %q not running on Node %q", args.AllocID, args.NodeID), false)
          		return nil
          	}
          	if alloc.TerminalStatus() {
          		setError(fmt.Errorf("Can't request Vault token for terminal allocation"), false)
          		return nil
          	}
          
          
	// Check if alloc has Vault
          	vaultBlocks := alloc.Job.Vault()
          	if vaultBlocks == nil {
          		setError(fmt.Errorf("Job does not require Vault token"), false)
          		return nil
          	}

(update: dropped the inline code paste - didn’t realize github code links were rendered inline some)

… makes me think that there needs to be either:

special case handling for post-stop tasks (however in this context - it is run to get tokens for all tasks … so would be very messy)
pre-fetch tokens for post-stop tasks (while keeping them alive) so that when the post-stop task comes around, the token is ready

roman-ttt · August 25, 2023, 9:00pm

Facing the same issue. In my case, access to vault is required for access to volumes where we need to archive logs after the run of the main task.

Topic		Replies	Views
Failures to derive Vault token Nomad	2	699	January 27, 2023
Error performing RPC to server: allocation is terminal Nomad	1	310	November 6, 2023
Authorization to use Jobs API within running Nomad task Nomad	3	419	January 12, 2022
All nomad job switched to pending state after vault outage Nomad vault	4	377	August 27, 2021
Issue on passing vault secrets to nomad Nomad vault	0	243	March 20, 2023

"Can't request Vault token for terminal allocation" when parsing template in poststop task

Related topics