Hello,
We have a nomad job that is acting as a service checker, it has a dummy docker image and runs a check on the service block, it also has an upstream to another service on port 18001. This check is a custom script written by us and runs every 60s. The script could be simply explained as a curl, which uses localhost:18001 as an argument.
The problem is that we are seeing high cpu spikes, well above what they should be. They are probably related to the check script because the spikes always occur in different minutes of the same second and because we tested without the check and it was working fine.
It’s also weird that some spikes are 100% some are 90% and some are 0% … it is not consistent.
We also see this error on the Nomad logs
2023-10-03T15:14:39.341Z [DEBUG] client.driver_mgr.exec.executor.nomad: time="2023-10-03T15:14:39Z" level=warning msg="cannot serialize hook of type configs.FuncHook, skipping": alloc_id=7896e801-5219-e023-c11b-83d51432d079e driver=exec task_name=service-checker
The job has a 500mhz cpu limit; is it likely that my job is using 100% of it for a simple curl script?
Thank you.
Hi @barrosoguillermo55,
The job has a 500mhz cpu limit; is it likely that my job is using 100% of it for a simple curl script?
There isn’t enough information here to understand what is exactly happening, but on the surface, yes, it looks like this job is spiking at points and using 100% of the CPU it has been allocated.
It’s also weird that some spikes are 100% some are 90% and some are 0% … it is not consistent.
I would argue that this pattern is expected for a process/application that is CPU sensitive or experiencing performance instability.
While you mention the script is like running CURL, is this a bash wrapper around that program, or written in a programming language which has a GC, for example. There are also details which complicate the situation such as the “dummy docker image” and Consul Connect involvement.
Thanks,
jrasell and the Nomad team
1 Like
Hi @jrasell, thanks for the quick reply!
This is a shell script written around the CURL program, it has a few lines with some if else logic but basically runs plain linux curl commands.
The dummy docker image is a simple Alpine image that executes the “tail -f /dev/null” command.
About the Connect involvement, there shouldn’t be any problem, we already have a lot of other containers and jobs running with these specs and it works just fine.
Thanks.