Translating docker-compose.yml with multiple services using a common image ... to Nomad Job

Nomad
1

Hi Everyone,

I am a huge fan of Nomad and recently converted from Kubernetes due to the fact that we are moving to Nomad at my job to run all of our production workloads. We also use Vault, Consul, and Terraform so we are a very Hashicorp centric company.

The problem I am running into is with creating the Nomad job file for the OWASP tool, “Defect Dojo.” I have scoured the docs and have found helpful things such as the task dependencies being achieved through the lifecycle stanza and the prestart statement, but where I am confused is with the fact that the docker-compose.yml file defines multiple services that all share the same image. Here is a link to the docker-compose.yml:

The services used are:

  • nginx (defectdojo/defectdojo-nginx:latest)
  • uwsgi (defectdojo/defectdojo-django:latest)
  • celerybeat (defectdojo/defectdojo-django:latest)
  • celeryworker (defectdojo/defectdojo-django:latest)
  • initializer (defectdojo/defectdojo-django:latest)
  • mysql (mysql/mysql:latest)

I also read on this forum that I could think of Nomad tasks as loosly equal to docker-compose services, so I took a crack at writing the Nomad job file. Any help would be greatly appreciated. This is what I came up with but it didn’t work: (I dont have the mysql component here because I already have it running fronted by Fabio – which is also why its defined on port 3308):

job "defectdojo" {
    datacenters = ["wdc"]
    type = "service"

    group "nginx" {
        network {
            port "nginx" {
                static = "8443"
            }
        }

        service {
            name = "nginx"
            port = "nginx"

            check {
                type = "tcp"
                port = "nginx"
                interval = "10s"
                timeout = "2s"
            }
        }

        task "initializer" {
            driver = "docker"
            config {
                image = "defectdojo/defectdojo-django:latest"
                entrypoint = ["/wait-for-it.sh", "mysql:3306", "--", "/entrypoint-initializer.sh"]
                mounts = [
                    {
                      type = "bind"
                      target = "/app/docker/extra_settings"
                      source = "./docker/extra_settings"
                    }
                ]
            }

            resources {
                cpu    = 200
                memory = 128
            }

            service {
                name = "initializer"
            }

            env {
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_ADMIN_USER="admin"
                DD_ADMIN_MAIL="first.last@company.com"
                DD_ADMIN_FIRST_NAME="Admin"
                DD_ADMIN_LAST_NAME="User"
                DD_INITIALIZE="true"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq}"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }
        }

        task "celeryworker" {
            driver = "docker"

            config {
                image = "defectdojo/defectdojo-django:latest"
                entrypoint = ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-celery-worker.sh"]
                mounts = [
                    {
                    type = "bind"
                    target = "/app/docker/extra_settings"
                    source = "./docker/extra_settings"
                    }
                ]
            }

            resources {
                cpu    = 200
                memory = 128
            }

            service {
                name = "celeryworker"
            }

            env {
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_CELERY_BROKER_USER="guest"
                DD_CELERY_BROKER_PASSWORD="guest"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }
        }

        task "celerybeat" {
            driver = "docker"
            config {

                entrypoint = ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-celery-beat.sh"]
                mounts [
                    {
                        type = "bind"
                        target = "/app/docker/extra_settings"
                        source = "./docker/extra_settings"

                    }
                ]

            }

            resources {
                cpu    = 200
                memory = 128
            }

            service {
                name = "celerybeat"
            }

            env {
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_CELERY_BROKER_USER="guest"
                DD_CELERY_BROKER_PASSWORD="guest"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }

        }

        task  "uwsgi" {
            driver = "docker"
            config {
                image = "defectdojo/defectdojo-django:${DJANGO_VERSION:-latest}"
                entrypoint: ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-uwsgi.sh"]
                mounts [
                    {
                        type = "bind"
                        target = "/app/docker/extra_settings"
                        source = "./docker/extra_settings"

                    }
                ]

            }

            resources {
                cpu    = 200
                memory = 128
            }

            service {
                name = "uwsgi"
            }

            env {
                DD_DEBUG="false"
                DD_DJANGO_METRICS_ENABLED="false"
                DD_ALLOWED_HOSTS=${DD_ALLOWED_HOSTS:"*"
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_CELERY_BROKER_USER="guest"
                DD_CELERY_BROKER_PASSWORD="guest"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }
        }

        task "nginx" {
            driver = "docker"
            config {
                image = "defectdojo/defectdojo-nginx:${NGINX_VERSION:-latest}"
                ports = ["nginx"]
            }
            env {
                NGINX_METRICS_ENABLED="false"
            }
        }
    }
  }
}
2 Likes
2

Hi @j3deye :wave: ,

I think a service would map better to a group in Nomad, since it’s at this level that things like:

are defined (among others). Being able to change these parameters for each component could be handy (for example, you might want to increase the number of celery workers).

Groups in Nomad can have multiple containers in them, which is not possible in a Docker Compose service, so I think your groups would only have one task (which is totally fine).

Perhaps the only exception would be for the initializer container, which seems to run some database migrations. Maybe it could be a prestart task inside your main app group (which I think is uwsgi?).

So your job would look something like this:

job "defectdojo" {
  group "nginx" {
    task "nginx" {...}
  }

  group "uwsgi" {
    task "initializer" {...}
    task "uwsgi" {...}
  }

  group "celerybeat" {
    task "celerybeat" {...}
  }

  group "celeryworker" {
    task "celeryworker" {...}
  }
}

From a quick look there are a few syntax errors in the job you posted:

  • DD_ALLOWED_HOSTS=${DD_ALLOWED_HOSTS:"*" is missing a closing }. This is also not valid Nomad syntax, I think you would want to use variables.
  • entrypoint: ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-uwsgi.sh"] is not valid HCL (it should be a = instead of :)
  • the task "uwsgi" { block is not properly closed, so it’s encapsulating the next task.
  • mounts [ is also not valid HCL

I would suggest that you start small and try to add one group at a time to your job, making sure you are able to at least run nomad job validate successfully before adding a new group.

Try to run a job with just the celeryworker group (it seems pretty self-contained) and let me know how it goes :grinning_face_with_smiling_eyes:

1 Like
3

@lgfa29

Thank you very much for your reply and counsel. I think I implemented the job file as you suggested but unfortunately no luck yet getting it to run correctly. Here is the revised job file:

job "defectdojo" {
    datacenters = ["wdc"]
    type = "service"

    group "nginx" {
        network {
            port "nginx" {
                static = "8080"
            }
        }

        service {
            name = "nginx"
            port = "nginx"

            check {
                type = "tcp"
                port = "nginx"
                interval = "10s"
                timeout = "2s"
            }
        }

        task "nginx" {
            driver = "docker"
            config {
                image = "defectdojo/defectdojo-nginx:latest"
                ports = ["nginx"]
            }
            env {
                NGINX_METRICS_ENABLED="false"
            }
        }
    }

    group "uwsgi" {
        # network {
        #     port "uwsgi" {
        #         static = "8443"
        #     }
        # }

        # service {
        #     name = "uwsgi"
        #     port = "uwsgi"

        #     check {
        #         type = "tcp"
        #         port = "uwsgi"
        #         interval = "10s"
        #         timeout = "2s"
        #     }
        # }

        task "initializer" {
            driver = "docker"

             lifecycle {
                hook = "prestart"
                sidecar = false
            }

            config {
                image = "defectdojo/defectdojo-django:latest"
                # ports = ["uwsgi"]
                entrypoint = ["/wait-for-it.sh", "mysql:3306", "--", "/entrypoint-initializer.sh"]
                mounts = [
                    {
                      type = "bind"
                      target = "/app/docker/extra_settings"
                      source = "./docker/extra_settings"
                    }
                ]
            }

            resources {
                cpu    = 200
                memory = 128
            }

            # service {
            #     name = "initializer"
            # }

            env {
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_ADMIN_USER="admin"
                DD_ADMIN_MAIL="first.last@company.com"
                DD_ADMIN_FIRST_NAME="Admin"
                DD_ADMIN_LAST_NAME="User"
                DD_INITIALIZE="true"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq}"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }
        }

        task  "uwsgi" {
            driver = "docker"
            config {
                image = "defectdojo/defectdojo-django:latest"
                # ports = ["uwsgi"]
                entrypoint = ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-uwsgi.sh"]
                mounts = [
                    {
                        type = "bind"
                        target = "/app/docker/extra_settings"
                        source = "./docker/extra_settings"

                    }
                ]

            }

            resources {
                cpu    = 200
                memory = 128
            }

            # service {
            #     name = "uwsgi"
            # }

            env {
                DD_DEBUG="false"
                DD_DJANGO_METRICS_ENABLED="false"
                DD_ALLOWED_HOSTS="*"
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_CELERY_BROKER_USER="guest"
                DD_CELERY_BROKER_PASSWORD="guest"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }
        }

    }

    group "celeryworker" {
        task "celeryworker" {
            driver = "docker"

            config {
                image = "defectdojo/defectdojo-django:latest"
                entrypoint = ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-celery-worker.sh"]
                mounts = [
                    {
                    type = "bind"
                    target = "/app/docker/extra_settings"
                    source = "./docker/extra_settings"
                    }
                ]
            }

            resources {
                cpu    = 200
                memory = 128
            }

            # service {
            #     name = "celeryworker"
            # }

            env {
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_CELERY_BROKER_USER="guest"
                DD_CELERY_BROKER_PASSWORD="guest"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }
        }
    }

    group "celerybeat" {
        task "celerybeat" {
            driver = "docker"
            config {
                image = "defectdojo/defectdojo-django:latest"
                entrypoint = ["/wait-for-it.sh", "mysql:3306", "-t", "30", "--", "/entrypoint-celery-beat.sh"]
                mounts = [
                    {
                        type = "bind"
                        target = "/app/docker/extra_settings"
                        source = "./docker/extra_settings"

                    }
                ]

            }

            resources {
                cpu    = 200
                memory = 128
            }

            # service {
            #     name = "celerybeat"
            # }

            env {
                DD_DATABASE_URL="mysql://defectdojo:defectdojo@mysql.service.consul:3308/defectdojo"
                DD_CELERY_BROKER_USER="guest"
                DD_CELERY_BROKER_PASSWORD="guest"
                DD_SECRET_KEY="hhZCp@D28z!n@NED*yB!ROMt+WzsY*iq"
                DD_CREDENTIAL_AES_256_KEY="&91a*agLqesc*0DJ+2*bAbsUZfR*4nLw"
            }

        }
    }
}
4

Great! So now it’s more of an application issue rather than a Nomad job file problem.

To help you with this I will need some runtime information. Do you see any errors in the allocation logs?