How to set Kernel parameters using sysctl for a Hashicorp Nomad job

Nomad
1

The docs says I can set the kernel parameters using sysctl for a docker task like so:

config {
  sysctl = {
    "net.core.somaxconn" = "16384"
  }
}

This indeed works. But when I tried,

     sysctl = {
          "net.core.somaxconn" = "16384"
          "net.core.rmem_default" = 134217728
          "net.core.rmem_max" = 134217728
          "net.core.wmem_default" = 134217728
          "net.core.wmem_max" = 134217728
          "vm.max_map_count" = 1000000
        }

I’m getting the following error.

Sep 28, '22 19:30:22 +0530 	
Driver Failure 	
Failed to start container fa2179c3fbfe0a216e457449cfb72a78e08c0be45f10ba9596004fbfc51e5cac: API error (400):
failed to create shim task: OCI runtime create failed: 
runc create failed: 
unable to start container process: 
error during container init: 
open /proc/sys/net/core/rmem_default: 
no such file or directory: unknown

I couldn’t find anywhere in the docs what are the allowed parameters to set using this config.

I spent the whole day banging my head on this issue.

Please let me know if any more info is needed.

In case you are curious I’m trying to run Solana devnet validator as a container in Nomad.

2

Why do you have quotes only around the first value?

Can you set these parameters when starting w/o Nomad?

3

The quotes doesn’t matter. I tried with them as well. Same error.

I haven’t tried without Nomad. Will try and let you know.

4

The sysctl values are specific to your kernel and what Docker supports. These values are merely passed through to Docker from Nomad. At a glance of docker run | Docker Documentation, it looks like net.* values are not supported when using host networking - is your task doing that?

If you can make the task work outside of Nomad, we should be able to understand what your job config needs to look like to make it work.

1 Like
5

A link to the docker docs in the nomad docs would be useful.

1 Like
6

Sorry for the late reply. No, I’m not running on host network. The network mode is bridge with CNI plugin.

I just want to merely expose those values for the Solana app to check them. The parameters are set on the host for it to run without any issues.

7

The only thing that worked for me was disabling the os limit checks of the application.

I think the only real option is to run the container in privileged mode and add sysctl commands to set these parameters in the entrypoint script as root user.