Is it possible to persist sysbatch job so newly registered clients also run it?

Nomad
1

My sysbatch job is purged after being ran on every nodes. I want the job to persist, so newly registered clients could run it. Is it possible?

2

I doubt that the logic would work like that.

The “machines to run on” would be matched when the job was submitted (evaluated).

Also the semantics of the “batch” part mean a “start” and an “end”, so once it ends, it wouldn’t run automatically on new clients.

At least, that is how I would think about this.

1 Like
3

Hi @qbx2,

Thanks @shantanugadgil your analysis is correct. Once a sysbatch job has been registered and gone through the scheduling process, the allocation placements will be considered the final placements of that job. Once the placed allocations have run to completion, the job will be considered completed, and will be garbage collected at the next interval.

What work is the sysbatch job performing that needs to be performed on each client that enters the cluster? Could this be solved by usisng the system scheduler, dispatch jobs, or even configuration management?

Thanks,
jrasell and the Nomad team

2 Likes
4

For example, I want to generate configuration files for nomad and execute ‘systemctl restart nomad’ using raw_exec to configure nomad remotely. I thought sysbatch because the job needs to executed just once for a node. At the moment, I am using system scheduler with infinite sleep loop at the end of job. Are there any better ways? Additionally, I wonder if we can use parameterized/periodic sysbatch for this case.

5

Yes, I believe using some form of parameterized job for be a good approach here. Potentially using batch and executing a job per new client that gets started and wants to join the cluster.

I am curious, what configuration files are generated that are not needed to allow the client to join the cluster in the first place, or that cannot be generated at instance bootstrap?

Thanks,
jrasell and the Nomad team

6

Because requirements can vary and there are services which need to avoid downtime as possible, I just wanted to dynamically change nomad configurations without rolling out the node (Think on-premise database servers).
In this setup, for example, I can configure plugin docker to allow NET_ADMIN and just restart nomad when I need it later.

7

hi @qbx2 just something to consider … in the past I have used the at command (and atd daemon) to schedule a restart “2 minutes later” for things which are restarting themselves.

i.e.:

  • the raw_exec runs, changes the config files.
  • part of the raw_exec script is to potentially validate the config, etc.
  • if validate, etc. looks good, then it “schedules” a systemctl restart nomad two minutes later.

This allows the currently running task to finish successfully, without the agent restart affecting the current run.

# just-a-thought :slight_smile:

1 Like
8

Any thoughts? I think it really would helpful if nomad supports it.

9

@qbx2,

If you believe Nomad should support persistent style sysbatch jobs then please feel free to open an issue against the GitHub repository. Technically this would be very challenging to implement as it is distinctly different to all other scheduling and reconciliation behaviour.

Personally, I feel the current flexibility with parameterized jobs, system, and sysbatch is sufficient and would be curious if you can make this work.

Thanks,
jrasell and the Nomad team