Nomad / Podman / Rhel8 Driver difficulties

Nomad
1

Hey guys,

Trying to POC using Nomad and Podman on a Rhel8 server.

Currently running a Vagrant file which is just spinning up a rhel8 server on my laptop.
I’ve manually installed Podman, and I’m trying to use nomad to orchestrate a container via Podman.
(running containers normally (rootless) works fine)

Should be fairly simple…

Following the following guides:

I’m running the nomad agent on my host with the podman plugin in the plugin dir:
sudo nomad agent -dev -bind 0.0.0.0 -plugin-dir=/opt/nomad/plugins/

I’ve made a file named ‘podman-example.nomad’ which contains:

job "redis" {
  datacenters = ["dc1"]
  type        = "service"

  group "redis" {
    network {
      port "redis" { to = 6379 }
    }

    task "redis" {
      driver = "podman"

        config {
          image = "docker://redis"
          ports = ["redis"]
        }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

Trying to run nomad job run podman-example.nomad, gives me the following in the agent logs:

Could not get podman info: driver=podman @module=podman err="Get "http://u/v1.0.0/libpod/info": dial unix /run/podman/podman.sock: connect: no such file or directory

Which I assume is a problem with varlink not working correctly, even though I’ve followed the steps here - Programmatic remote access to Podman via the varlink protocol

I was wondering if anyone has had success getting nomad to orchestrate containers in Redhat 8.

Thanks

2

Hi @Francola, what version of podman are you using? Varlink has been deprecated and removed in recent versions of Podman and the latest nomad-driver-podman now supports podmans HTTP API.

You can read more about Podmans varlink removal here Podman API v1.0 Deprecation and Removal Notice

I think you’ll want to ensure you have systemd units configured for a podman service and socket. There is some info on that here Improved systemd integration with Podman 2.0.

The following should work as a way to test that that the driver can reach out to the podman api sudo curl -v -s --unix-socket /run/podman/podman.sock http://d/v1.0.0/libpod/info

3

I’m having the a very similar problem (minus the vargrant and Varlink).

I’m on a RHEL 8 try to build a POC using nomad and podman and I’m getting the same error @Francola is getting.

But it might be a podman problem since i can’t get the command suggested by @drewbailey to succeed.

I’ve opened an issue for podman here: failure when running `sudo curl -v -s --unix-socket /run/podman/podman.sock http://d/v1.0.0/libpod/info` · Issue #10714 · containers/podman · GitHub

4

If this isn’t sorted out yet… I ran into same issue with Nomad and Podman.
We need to enable Podman API for Nomad to talk to Podman.
Here’s more…
Sneak peek: Podman’s new REST API | Enable Sysadmin (redhat.com)
In summary… I ran the below commands to get it working.

We can do it in two ways.
Temporary:
Running the below commands to start Podman service with a set duration.

# podman system service -t 0 &

or

# podman system service -t 5000 &

Permanent:
Below package is needed for the API Service.

# yum install podman-remote

This is just an extra step to make sure the contents are right in the file.

# cat /usr/lib/systemd/system/podman.socket

Commands to enable the daemon

# systemctl daemon-reload
# systemctl enable --now podman.socket

After all this is done, if you still end up with an error, make sure sock file exists.

# ls -lt /run/podman/podman.sock

If you don’t see it…

# touch /run/podman/podman.sock

And retry. It should work. Because it’s working for me. :slight_smile:

2 Likes
5

Not directly related… but would be good to know if there are plans around enhancing Nomad & Podman in the light of the recent IBM-HashiCorp acquisition bringing both Red-Hat (Podman) and Hashicorp (Nomad) under the same umbrella…