Hi,
I’m working my way through the Nomad learning material. Even though moving my app to Nomad is still far off I can start using Nomad to scale Gitlab CI/CD almost immediately. Has anyone done this? Please share tips/resources/war stories etc.
Thanks,
Manoj
Hi @egmanoj ,
Thanks for using Nomad!
I haven’t done this personally, so no war stories, but I’ve seen a few articles about it, so here are some resources for you. My disclaimer is I haven’t read or watched all of these entirely, so they might or might not be 100% your use case. I hope some of these prove useful. Let us know how it goes!
https://www.burgundywall.com/post/continuous-deployment-gitlab-and-nomad
Hi @DerekStrickland,
Thanks for the links. Went over them in detail. They mostly cover using Nomad with CI/CD - a very important topic. Unfortunately in my case I’m not quite there just yet.
I am hoping to become familiar with the Nomad ecosystem by scaling a Gitlab CI/CD pipeline using Nomad. This means running gitlab-runner in Nomad, and using Nomad Autoscaler (+AWS) to spin CI/CD runner nodes up/down based on load. Chris Baker’s talk that you linked to mentions running gitlab-runner using Nomad but doesn’t provide any details.
Thanks again.
Manoj
Hi @egmanoj,
I have plans do go after a gitlab runner custom executor: so a runner running (on nomad) that creates nomad (docker) jobs for executing the CI job. quite similar the custom executor for lxd (Using LXD with the Custom executor | GitLab) was made.
Currently we run a few static gitlab runners services with docker.socket:
job:
variable "datacenters" {
type = list(string)
default = ["dc1"]
}
variable "cpu" {
type = number
default = 1000
}
variable "memory" {
type = number
default = 1024
}
variable "config" {
type = string
}
variable "image" {
type = string
default = "gitlab/gitlab-runner:alpine"
}
variable "ca_cert" {
type = string
}
job "gitlab-runner-1" {
datacenters = var.datacenters
type = "service"
constraint {
attribute = "${node.class}"
value = "workers"
}
group "gitlab-runner" {
count = 1
ephemeral_disk {
size = 1000
}
task "gitlab-runner" {
template {
change_mode = "noop"
destination = "local/gitlab-runner-config.toml"
data = var.config
}
template {
change_mode = "noop"
destination = "local/ca.crt"
data = var.ca_cert
}
driver = "docker"
config {
image = var.image
volumes = [
"/var/run/docker.sock:/var/run/docker.sock",
"local/gitlab-runner-config.toml:/etc/gitlab-runner/config.toml",
"/root/.docker/config.json:/root/.docker/config.json",
"local/ca.crt:/etc/gitlab-runner/certs/ca.crt",
]
}
resources {
cpu = var.cpu
memory = var.memory
}
}
}
}
config.hcl
config = <<EOT
concurrent = 2
check_interval = 0
log_format = "runner"
[session_server]
session_timeout = 1800
[[runners]]
name = "Nomad GitLab Runner 1"
limit = 0
output_limit = 4096
url = "https://gitlab.example.com"
environment = []
token = "..."
executor = "docker"
tls-ca-file = "/etc/gitlab-runner/certs/ca.crt"
[runners.docker]
privileged = true
"allowed_images" = [
"alpine:*",
"ruby:*",
"python:*",
"php:*",
"maven:*",
"node:*",
"busybox:*",
"docker:*",
"quay.io/ansible/toolset:*",
"mysql:*",
"postgres:*",
"hashicorp/*",
"registry.gitlab.com/pages/*",
"caddy:*",
"minio/mc:*",
]
dns = ["10.10.10.10", "10.10.10.11"]
memory = "1024m"
image = "alpine"
disable_entrypoint_overwrite = false
oom_kill_disable = true
disable_cache = false
volumes = [
"/var/run/docker.sock:/var/run/docker.sock",
"/root/.docker/config.json:/root/.docker/config.json:ro",
"/etc/gitlab-runner/certs/ca.crt:/etc/gitlab-runner/certs/ca.crt:ro",
"/cache",
]
shm_size = 0
[runners.docker.sysctls]
"net.ipv4.ip_forward" = "1"
EOT
I very recently deployed a custom executor for scaling GitLab CI/CD in my homelab for this 
I’ve mostly been doing this to try out using AppRole in vault however in the process I ended up writing a custom executor for dispatching a parametrised job and running the CI/CD stages inside of that (also very similar to the LXD example in the GitLab Docs)
I wouldn’t recommend it for any real production loads, mostly because its me messing around in my homelab and i’m still trying to determine if this is even a reasonable way of doing it, but if it helps as an example then the code for it can be found:
Static runners which dispatch the executors: hQ / Home Lab / CICD GitLab Runner · GitLab
Executor docker image: hQ / Home Lab / CICD GitLab Executor · GitLab
Executor parametised job definition: https://gitlab.com/hq-smarthome/home-lab/cicd-gitlab-linux-amd64-executor
Just as a follow up, I forgot that I planned to move these repositories so the new link to find all of them (and future ones) will be: hQ - CICD GitLab Group.
It would not let me edit my previous message 
…
@CarbonCollins
This seems super cool, also attempting homelab stuff fully utilizing hashi stuff, using ceph/amd64 and ARM. Your usage of consul connect has helped me a ton in the last couple of days since finding this.
CI/CD has been kinda far down the task list, was more kinda set up for using terraform and this. but was attempting to try out the above and replicate but running into issues with the approle stuff potentially, if you get a chance to document those vault policies I would super appreciate it
I’m glad you have been getting some use from the repos! It’s why I have them public in the first place
In regards to the Vault policies they are documented just not in a public repo yet (I will also admit there are quite a few repos that I have not published yet…)
The jist is the runner has something akin to:
path "auth/${executorBackendPath}/role/+/secret*" {
capabilities = ["create", "read", "update"]
min_wrapping_ttl = "10s"
max_wrapping_ttl = "2m"
}
which is deployed using Terraform with:
resource "vault_policy" "cicd_gitlab_runner" {
name = "cicd-gitlab-runner"
policy = templatefile("vault/policies/cicd-gitlab/runner.template.hcl", {
executorBackendPath: vault_auth_backend.cicd_gitlab_executor.path
})
}
And for the executor its:
# Allow tokens to look up their own properties
path "auth/token/lookup-self" {
capabilities = ["read"]
}
path "nomad/creds/cicd-gitlab-executor" {
capabilities = ["read"]
}
which allows the executor obtain a nomad token for deployment using: Generate Nomad Tokens with HashiCorp Vault | Nomad - HashiCorp Learn.
I will re-iterate my disclaimer though that I wouldn’t recommend it for any real production loads, mostly because its me messing around in my homelab :P. Speaking of which feel free to DM me if you have other questions regarding the HomeLab
Going back to the AppRole setup I’ve not made all the policies and such public yet as I’m not fully satisfied in what I previously linked and have started looking into seeing if replacing the AppRole setup with something similar to the JWT example from GitLab is a better route as it looks to allow restricting policies at a repo level: Using external secrets in CI | GitLab (specifically the CI_JOB_JWT portion)
It’s a little slow going to look into this all as I only really get to work on it in the evenings or weekends but those repos will be updated as I go