How to share data Nginx between and app container

Hello,

I’m evaluating Nomad to use it instead of Docker Swarm and have most topics done so far, like Consul and Vault. Now I try to get a simple project converted from a docker-composer file, to nomad, but I stuck at one point:

This project (Django) generates (migrate static … ) files, which I need also on the Nginx container. On the compose file, I use the same Docker volume, but how can I achieve the same with Nomad ? I red a lot about CSI / NFS / alloc/ … but this sounds pretty complicated, as these data may obsolete with the next image version.
It would be no issue, if I can stick the Nginx and the app container on the same host, and make sure, if the containers are stopped, the shared volumes needs to be deleted too.

So, the Django container starts, generates the files (css and other static things) and Nginx should deliver these … Is that possible ?

cu denny

If the files are generated on every start, and django/nginx run in the same job-group, just use the alloc folder. It is shared between all tasks in a group. It can also be made semi-persistent using the ephemeral disk config, if you like.

Hello @runeron

thanks for the reply. I’ve searched a while and try to understand, how this /alloc part works.

My job config looks like this (omitted themplates and not required sections):

job "backoffice_gunicorn" {

# We want it on fra-test
   datacenters = ["fra-test"]

# First we group the Gunicorn backoffice
      group "django" {
         network {
            port "api" {
               to = 8000
            }
         }

         service {
            name = "django"
               tags = ["backoffice","django"]
               port = "api"

               check {
                  type = "tcp"
                     port = "api"
                     interval = "10s"
                     timeout = "2s"
               }
         }

         task "django" {
            driver = "docker"

            config {
               image = "fra-test-harbor.example.local/testing/backoffice/test:latest"
                  ports = ["api"]
                  volumes = ["/alloc/data/backoffice:/app/static/"]
            }

            resources {
               cpu = 1000
                  memory = 256
            }
         }
      }


# Webserver part
   group "nginx" {
      network {
         port "https" {
            to = 443
         }
         port "api" {
            to = 8000
         }
      }

      service {
         name = "nginx"
            tags = ["backoffice","nginx"]
            port = "api"

            check {
               type = "tcp"
                  port = "https"
                  interval = "10s"
                  timeout = "2s"
            }
      }

      task "nginx" {
         driver = "docker"

         config {
            image = "fra-test-harbor.example.local/testing/backoffice/nginx:latest"
               ports = ["https"]
               volumes = ["/alloc/data/backoffice:/home/app/static/","certs/:/certs"]
         }

         resources {
            cpu = 1000
               memory = 256
         }
      }
   }
}

So, if I understand you right … I have to drop the nginx group and move all parts to the top, but I never saw examples, with more then one container on a task.

Maybe, I was not clear enough: The Django (gunicorn) is a container which is the backoffice app. Nginx is the reverse proxy for Gunicorn, but Nginx should deliver all the static files to avoid too much noise on the Gunicorn process. On the first start from the backoffice container, it will generate static files (python3 manage.py collectstatic) on /app/static/. These files needs to be on the Nginx containers too, under /home/app/static/.

The docker-compose file parts:

...
          services:
            backoffice_nginx:
              image: "{{BACKOFFICE_NGINX_IMAGE}}"
              hostname: localhost
              restart: always
              container_name: backoffice_nginx
              volumes:
                - backoffice-gunicorn_data:/home/app/static/
                - certs:/certs
              networks:
...
            backoffice_gunicorn:
              image: "{{BACKOFFICE_IMAGE}}"
              restart: always
              container_name: backoffice_gunicorn
              hostname: backoffice
              volumes:
                - backoffice-gunicorn_data:/app/static/
...

I red also things about using the sidecar stanza … but also never saw good examples, so that Nginx and the app containers runs always on the same node and can share the same files.

So, do you have examples for it … it would be very helpful :slight_smile:

cu denny

hi,

ah, I thought … I’ve found the issue … but both containers are on the same job … but how can they access the same /alloc … if they run on different hosts ?

Ok, it seems, I have to change the job from two groups to one.

Same group = same host (and same alloc folder). Separate groups might just as well be separate jobs, and it would be the same.

If you want to split the jobs, then you need some sort of common storage. E.g. CSI as you mentioned.

It’s quite common to have multiple containers (tasks) in same group. E.g. init containers, logging sidecars and so on.

I’d think you could just use a single group with both tasks, and that would be fine. Just configure to use the correct alloc in tasks, or use the mount option to map it to the folder you want for each task.

On mobile now, so no examples at hand. If you check out the nomad-pack registry, you can probably find some example jobs. Then you can render a jobfile & inspect.

Hi @runeron

I’ve got it working ! Thanks so much.

It’s from the Consul view still not perfect … but it works, thats the important thing. The hard part was, to get all the small parts / pieces together as no one wrote a complete example. I will do that, as I’m sure, that I’m not the only one, who had this problem and wasted days to solve it.

Also using for example Minio as storage … is not an trivial thing … as docs are more or less does not exists, besides pre / testing on Github issues …

  • My example below does:
    • Create three containers (Nginx as reverse / prepare statics / Gunicorn Django)
    • Get values for ENV inside the containers
    • Get certs and secrets from Vault
    • Share files, which was created on a different tasks (thanks to Daniela)

My full example is:

job "backoffice_gunicorn" {

# For our secrets
   vault {
      policies = ["access-tables"]
   }

# We want it on fra-test
   datacenters = ["fra-test"]

      group "backoffice" {
         network {
            port "api" { to = 8000 }
            port "https" { to = 443 }
         }

    ephemeral_disk {
      migrate = false
      size    = 300
      sticky  = false
    }

         service {
            name = "django"
               tags = ["backoffice","django"]
               port = "api"
               check {
                  type = "tcp"
                     port = "api"
                     interval = "10s"
                     timeout = "2s"
               }
         }

         service {
            name = "nginx"
               tags = ["backoffice","nginx"]
               port = "api"

               check {
                  type = "tcp"
                     port = "https"
                     interval = "10s"
                     timeout = "2s"
               }
         }

         task "django-collectstatic" {
            lifecycle {
               hook = "prestart"
               sidecar = false
            }
            driver = "docker"
               template {
                  data = <<EOH
                  {{key "nomad/backoffice/environment"}}
                  EOH
                     destination = "local/env"
                     env         = true
               }

            template {
               data = <<EOH
               {{with secret "kv/docker/nomad/backoffice/secrets"}}
               {{range $key, $value := .Data.data}}
               {{$key}}={{$value}}{{end}}
               {{end}}
               EOH
                  destination = "secrets/file.env"
                  env = true
            }

            config {
               image = "fra-test-harbor.example.local/testing/backoffice/test:latest"
                  ports = ["api"]
                  entrypoint = [ "./docker-entrypoint.sh", "collectstatic" ]
                  force_pull = true
                  volumes = [ "${NOMAD_ALLOC_DIR}/data/backoffice/static:/app/static"]
                  auth {
                     username = "robot$devops"
                        password = "secret"
                  }
            }

            resources {
               cpu = 1000
                  memory = 256
            }
         }

         task "django" {
            driver = "docker"
               template {
                  data = <<EOH
                  {{key "nomad/backoffice/environment"}}
                  EOH
                     destination = "local/env"
                     env         = true
               }

            template {
               data = <<EOH
               {{with secret "kv/docker/nomad/backoffice/secrets"}}
               {{range $key, $value := .Data.data}}
               {{$key}}={{$value}}{{end}}
               {{end}}
               EOH
                  destination = "secrets/file.env"
                  env = true
            }

            config {
               image = "fra-test-harbor.example.local/testing/backoffice/test:latest"
                  ports = ["api"]
                  volumes = [ "${NOMAD_ALLOC_DIR}/data/backoffice/static:/app/static"]
                  auth {
                     username = "robot$devops"
                        password = "secret"
                  }
            }

            resources {
               cpu = 1000
                  memory = 256
            }
         }

         task "nginx" {
            driver = "docker"

               template {
                  data = <<EOH
                  {{- if service "django" -}}
                  {{- with index (service "django") 0 -}}
                  API={{ .Address }}:{{ .Port }}
                  {{- end -}}
                  {{- end }}
                  EOH

                     destination = "local/envvars.txt"
                     env = true
               }

            template {
               data = <<EOH
               {{key "nomad/backoffice/environment"}}
               EOH
                  destination = "local/env"
                  env         = true
            }

            template {
               data        = "{{ with secret \"kv/docker/nomad/certs\"  }}{{.Data.data.key}}{{end}}"
                  destination = "certs/server.key"
                  change_mode = "restart"
                  perms        = "0644" 
                  splay       = "1m"
            }

            template {
               data        = "{{ with secret \"kv/docker/nomad/certs\"  }}{{.Data.data.crt}}{{end}}"
                  destination = "certs/server.crt"
                  change_mode = "restart"
                  perms        = "0644" 
                  splay       = "1m"
            }
            template {
               data        = "{{ with secret \"kv/docker/nomad/certs\"  }}{{.Data.data.dh}}{{end}}"
                  destination = "certs/dh.pem"
                  change_mode = "restart"
                  perms        = "0400" 
                  splay       = "1m"
            }
            config {
               image = "fra-test-harbor.example.local/testing/backoffice/nginx:latest"
                  ports = ["https"]
                  volumes = [
                     "${NOMAD_ALLOC_DIR}/data/backoffice/static:/home/app/static/",
                     "certs/:/certs"
                  ]
                  auth {
                     username = "robot$devops"
                        password = "secret"
                  }
            }

            resources {
               cpu = 1000
                  memory = 256
            }
         }

      }
}