Driver Docker: failed to create shim task

Hello,

Thank you for reading.

I am trying to get NearBeach up and running under nomad however I keep hitting some hurdles along the way. I am currently only running nomad under a dev environment as I develop a way to get this working.

My current nomad file is setup as such

job "prod.nearbeach" {
  datacenters = ["dc1"]

  group "nearbeach" {
    task "nearbeach" {
      driver = "docker"

      config {
	image = "robotichead/nearbeach:latest"
	port_map {
          webserver = 8000
	  smtp = 2525
	  db = 3306
        }
 	command = "/bin/bash"
        args = ["-c", "python manage.py wait_for_database && ls -al && python manage.py migrate && python manage.py initadmin && python manage.py runserver 0.0.0.0:8000"]
      }

      env {
          SECRET_KEY = "iPprijieaZs1wcZ4"
          SMTP_EMAIL_HOST = "smtp.mailtrap.io"
          SMTP_EMAIL_PORT = "2525"
          SMTP_EMAIL_HOST_USER = "ba914cbac41ed1"
          SMTP_EMAIL_HOST_PASSWORD = "cada0e56f86350"
          MYSQL_DATABASE = "NearBeach"
          MYSQL_USER = "nearbeach"
          MYSQL_PASSWORD = "Test1234_"
          MYSQL_HOST = "192.168.64.7"
          ADMIN_USERNAME = "luke"
          ADMIN_EMAIL = "luke@nearbeach.org"
      }

      resources {
        cpu    = 500
        memory = 256
        network {
          mbits = 10
	  port "webserver" {}
          port "smtp" {}
          port "db" {}
        }
      }

      service {
        name = "redis-cache"
        tags = ["global", "cache"]
        port = "db"
        check {
          name     = "alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

After setting up the file, I will run the command nomad job run nearbeach.nomad and I’ll get the following results

==> Monitoring evaluation "ad8a1fb5"
    Evaluation triggered by job "prod.nearbeach"
    Allocation "ebeb718b" created: node "4e4db64d", group "nearbeach"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "ad8a1fb5" finished with status "complete"

The status of the job will stay pending. However when I look at the terminal running the nomad dev environment, I spot the following error

    2022/06/25 12:51:52.921134 [ERR] driver.docker: failed pulling container robotichead/nearbeach:latest: API error (500): Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
    2022/06/25 12:51:52.921678 [WARN] client: error from prestart: failed to initialize task "nearbeach" for alloc "5623d184-b1d1-afc4-48e9-cb96035fa857": Failed to pull `robotichead/nearbeach:latest`: API error (500): Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
    2022/06/25 12:51:52.921774 [INFO] client: Not restarting task: nearbeach for alloc: 5623d184-b1d1-afc4-48e9-cb96035fa857 
    2022/06/25 12:51:52.922438 [INFO] client.gc: marking allocation 5623d184-b1d1-afc4-48e9-cb96035fa857 for GC

I’ve been able to run NearBeach in a docker-compose container easily. With the following details

version: '3'

services:
    nearbeach:
        image: robotichead/nearbeach:latest
        container_name: nearbeach
        environment:
        - SECRET_KEY=MoreSecrets
        - EMAIL_HOST=smtp.mailtrap.io
        - EMAIL_HOST_USER=FakeUser
        - EMAIL_HOST_PASSWORD=NotARealPassword
        - EMAIL_PORT=2525
        - MYSQL_DATABASE=NearBeach
        - MYSQL_USER=nearbeach
        - MYSQL_PASSWORD=Test1234_
        - MYSQL_HOST=192.168.64.7
        - ADMIN_USERNAME=admin
        - ADMIN_EMAIL=fake-email@nearbeach.org
        volumes:
        - .:/ceansuite
        ports:
        - 8000:8000
        - 2525:2525
        command: >
            sh -c "python manage.py wait_for_database &&
                ls -al &&
                python manage.py migrate &&
                python manage.py initadmin &&
                python manage.py runserver 0.0.0.0:8000"
        restart: unless-stopped

I’ve essentially made sure it works in docker-compose first before copying all the env variables over into the nomad file.

I’m scratching my head at what to do next.

System Detail
I am running on a VM using multipass on a MacBook Pro (13-inch, M1, 2020).

Extra Info
NearBeach docker source - GitHub - robotichead/nearbeach-docker: A simple Docker file to install NearBeach

If you require any other information, please ask :slight_smile:

A few thoughts:

  • have you tried to replace your custom image with a common one, like Nginx, to check networking? You mention running in a VM, and wondering whether it gets a route to the internet.
  • is your image published to Hub? If not mistaken Nomad will try to get the image from there by default, unless you have a local registry.
  • the error is a name resolution failure. This points to another aspect of networking: Assuming the Hub is available. I worked last week with it, and pretty shifty, even buggy. Until it starts working Perhaps as simple as trying again?

Overall it seems your job definition is fine (I could not try it yet, but syntax and settings look ok). The error log is really on networking.

Thank you for your reply.

I’ve changed the docker image to “nginx”. It downloaded and start the server perfectly fine. Was able to get a “running” status. I however was not able to connect to the nginx server through the web browser - however I think this is a different issue.

Yes - NearBeach is published here - Docker Hub

Sadly I have been getting the same issue - even after testing nginx.

Thank you for your reply.

Hi @robotichead, sorry you’re having trouble getting started. Like @_ic pointed out this seems rooted in a networking issue, in that the Nomad Client’s docker library is unable to resolve registry-1.docker.io.

Can you share the output of dig registry-1.docker.io when run from inside the VM that the Nomad Client is running in?

Hello, I’ve run the command and obtained the following.

ubuntu@nomad:~/nomad-test$ dig registry-1.docker.io

; <<>> DiG 9.16.1-Ubuntu <<>> registry-1.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56434
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;registry-1.docker.io.		IN	A

;; ANSWER SECTION:
registry-1.docker.io.	77	IN	A	52.203.217.46
registry-1.docker.io.	77	IN	A	34.230.238.103
registry-1.docker.io.	77	IN	A	34.237.170.112
registry-1.docker.io.	77	IN	A	3.215.210.111
registry-1.docker.io.	77	IN	A	34.206.128.73
registry-1.docker.io.	77	IN	A	54.146.243.62
registry-1.docker.io.	77	IN	A	34.237.163.104
registry-1.docker.io.	77	IN	A	34.203.135.183

;; Query time: 72 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 29 19:48:04 AEST 2022
;; MSG SIZE  rcvd: 177