Hello all,
I am trying to get a better grasp on how Job scheduling in Nomad works, particularly how Nomad evaluates the network resources that a node has available. At a high-level the question boils down to:
Is Nomad aware of network resources (i.e. reserved ports) allocated on a Node outside of Nomad, or does it only track resources that it has allocated on a node itself?
In more detail, our use case involves operating Nomad alongside an older job orchestrator on the same nodes. The goal is for Nomad to eventually entirely replace the old orchestrator. The jobs have dependencies on static ports, unfortunately changing this is outside the scope of this migration effort. If a node has a job allocated on it by the old orchestrator and Nomad tries to allocate the same job on the same node during the migration, it will fail to bring the job online due to a port conflict.
I see that Nomad does evaluate reserved ports when it is evaluating a plan. The EvaluatePool will call down to evaluateNodePlan, which will eventually call AllocsFit to evaluate if a job will “fit” on a node. At this point, it will populate a network index of the node, which I see contains a “UsedPorts” field that will be checked against and throw an error if there’s a port conflict. What I just can’t figure out is how the network index is populating used ports.
I see that the collision checking happens by way of AddReservedPorts, which will use a helper called getUsedPortsFor to populate a bitmap with a Node’s used ports. But, this getUsedPortsFor helper is where I lose things.
How is it collecting that information from the Node? Is it using system level APIs? And if so where?
If it is in fact collecting information from the Node itself via some system level API then operating Nomad alongside the old orchestrator during the migration presents no issue. But, if this used ports field is populated only with a world-view scoped to ports that Nomad itself has allocated on a given node, then we will need to find some workaround to this issue.
Thanks all!