Nomad agent failing all of a sudden with errors pointing to nvidia

Nomad
1

Hi all, I used to run this script that would launch nomad and some other stuff and it worked fine until yesterday. Now everything looks normal until gets to the line where it runs nomad agent -dev. Running this command now outputs the following:

==> Starting Nomad agent...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x11 pc=0x4066ce]

goroutine 1 [running]:
github.com/NVIDIA/gpu-monitoring-tools/bindings/go/nvml._Cfunc_GoString(...)
        _cgo_gotypes.go:170
github.com/NVIDIA/gpu-monitoring-tools/bindings/go/nvml.errorString(0xc000000011, 0xc000a1f200, 0x41253b)
        github.com/NVIDIA/gpu-monitoring-tools@v0.0.0-20180829222009-86f2a9fac6c5/bindings/go/nvml/bindings.go:56 +0x54
github.com/NVIDIA/gpu-monitoring-tools/bindings/go/nvml.init_(0x150, 0x29026c0)
        github.com/NVIDIA/gpu-monitoring-tools@v0.0.0-20180829222009-86f2a9fac6c5/bindings/go/nvml/bindings.go:65 +0x73
github.com/NVIDIA/gpu-monitoring-tools/bindings/go/nvml.Init(...)
        github.com/NVIDIA/gpu-monitoring-tools@v0.0.0-20180829222009-86f2a9fac6c5/bindings/go/nvml/nvml.go:251
github.com/hashicorp/nomad/devices/gpu/nvidia/nvml.(*nvmlDriver).Initialize(...)
        github.com/hashicorp/nomad/devices/gpu/nvidia/nvml/driver_linux.go:9
github.com/hashicorp/nomad/devices/gpu/nvidia/nvml.NewNvmlClient(0x7f4e3859caa8, 0x150, 0x150)
        github.com/hashicorp/nomad/devices/gpu/nvidia/nvml/client.go:68 +0x28
github.com/hashicorp/nomad/devices/gpu/nvidia.NewNvidiaDevice(0x30b4558, 0xc000124440, 0x30e7cb8, 0xc0007264b0, 0xc00035e000)
        github.com/hashicorp/nomad/devices/gpu/nvidia/device.go:109 +0x34
github.com/hashicorp/nomad/devices/gpu/nvidia.glob..func1(0x30b4558, 0xc000124440, 0x30e7cb8, 0xc0007264b0, 0x0, 0x0)
        github.com/hashicorp/nomad/devices/gpu/nvidia/device.go:47 +0x49
github.com/hashicorp/nomad/helper/pluginutils/loader.(*PluginLoader).initInternal(0xc000726510, 0xc0007262d0, 0xc000726540, 0x0, 0x0, 0x0)
        github.com/hashicorp/nomad/helper/pluginutils/loader/init.go:96 +0x1dd
github.com/hashicorp/nomad/helper/pluginutils/loader.(*PluginLoader).init(0xc000726510, 0xc000a1f6c0, 0x2, 0x2)
        github.com/hashicorp/nomad/helper/pluginutils/loader/init.go:59 +0x87
github.com/hashicorp/nomad/helper/pluginutils/loader.NewPluginLoader(0xc000a1f6c0, 0x30ec578, 0xc0009a0f90, 0x30e7cb8)
        github.com/hashicorp/nomad/helper/pluginutils/loader/loader.go:135 +0x45d
github.com/hashicorp/nomad/command/agent.(*Agent).setupPlugins(0xc0004181e0, 0xc000311500, 0x0)
        github.com/hashicorp/nomad/command/agent/plugins.go:27 +0x15f
github.com/hashicorp/nomad/command/agent.NewAgent(0xc000313600, 0x30ec578, 0xc0009a0f90, 0x3070ac0, 0xc0007762d0, 0xc0007c7d60, 0x0, 0x0, 0x2569200)
        github.com/hashicorp/nomad/command/agent/agent.go:134 +0x1fb
github.com/hashicorp/nomad/command/agent.(*Command).setupAgent(0xc000967ce0, 0xc000313600, 0x30ec578, 0xc0009a0f90, 0x3070ac0, 0xc0007762d0, 0xc0007c7d60, 0x0, 0x2)
        github.com/hashicorp/nomad/command/agent/command.go:480 +0xb0
github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc000967ce0, 0xc00004e1a0, 0x1, 0x1, 0x0)
        github.com/hashicorp/nomad/command/agent/command.go:672 +0x4cc
github.com/mitchellh/cli.(*CLI).Run(0xc000964b40, 0xc000964b40, 0xc00012dc98, 0x37)
        github.com/mitchellh/cli@v1.1.0/cli.go:260 +0x41a
main.RunCustom(0xc00004e190, 0x2, 0x2, 0xc000060598)
        github.com/hashicorp/nomad/main.go:142 +0x4a7
main.Run(...)
        github.com/hashicorp/nomad/main.go:87
main.main()
        github.com/hashicorp/nomad/main.go:83 +0x65

I don’t care about the GPU right now, so I tried putting this in my nomad.hcl:

plugin "nvidia-gpu" {
  config {
    enabled = false
  }
}

But that doesnt seem to have solved it either. I tried un- and then re-installing nomad. No luck there either. I have no idea why this thing stopped working.

2

Hi @astolman,

Would you be able to provide the script you are using along with version information for Nomad?

Thanks,
jrasell and the Nomad team

3

The nomad version is 1.1.6, but I’ve tried rolling back all the way to 1.1.3 and that didn’t work. I’m not sure the rest of the script is relevant since this happens when I call nomad from the command line all on its own. It basically just launches a docker container first, and then brings up nomad.