We have our services and main application all in Kubernetes, but part of our system offers workers to run workloads on. This used to be all managed directly in Kubernetes, but we want to be able to run on “remote” machines too. So the idea of orchestrating through Nomad came up.
We have a POC running in 2 VM instances with a server on each. But I want to be able to scale up servers as needed easily and just keep our infra in Kubernetes if possible.
I did find the nomad-on-kubernetes repository and I followed parts of it. But from my understanding we don’t really need Consul or Vault, just Nomad, so I just did Nomad sections.
I get warnings in the logs
!! Running Nomad clients inside Docker containers is not supported. !!
!! Refer to https://www.nomadproject.io/s/nomad-in-docker for more information. !!
!! Set the NOMAD_SKIP_DOCKER_IMAGE_WARN environment variable to skip this warning. !!
I thought this only applied to clients and not servers, since the URL mentioned there says
While it’s possible to run Nomad servers inside containers, Nomad clients require extensive access to the underlying host machine
And I have been also trying to get the auto joining to work for k8s from go-discover (GitHub - hashicorp/go-discover: Discover nodes in cloud environments) but I get
2023-10-23T18:57:01.924Z [ERROR] agent.joiner: determining join addresses failed: error="discover: unknown provider k8s"
Could someone point me to how to get k8s provider to be recognized in go-discover.
And also, is doing this a bad idea for Nomad? Is it better to manage the machines manually? I would really like to keep it in Kubernetes.