KeyDB Deployment and Secure Password Rotation with HashiCorp Vault and Nomad

Nomad
,
1

As a newcomer to Nomad, I am eager to learn from the community and welcome any feedback on this job configuration. If you have suggestions for improvements, best practices, or optimization tips, please feel free to share them.

keydb.nomad.hcl

job "keydb-vanilla" {
  region      = "global"
  datacenters = ["dc1"]
  type        = "service"

  group "keydb" {
    count = 1

    vault {
      policies = ["nomad-keydb"]
    }

    network {
      mode = "host"
      port "redis" {
        static       = 6379
        to           = 6379
        host_network = "tailscale"
      }
    }

    update {
      min_healthy_time  = "10s"
      healthy_deadline  = "10m"
      progress_deadline = "20m"
      auto_revert       = true
    }

    service {
      name     = "keydb"
      port     = "redis"
      tags     = ["database", "cache"]
      provider = "nomad"
    }

    restart {
      attempts = 2
      interval = "30m"
      delay    = "15s"
      mode     = "fail"
    }

    task "keydb" {
      driver = "exec"
      user   = "keydb"

      config {
        command = "/usr/bin/keydb-server"
        args = [
          "/etc/keydb/keydb.conf",
        ]
      }

      resources {
        cpu    = 10000
        memory = 32 * 1024
      }

      template {
        data          = file("keydb.conf.tpl")
        destination   = "/etc/keydb/keydb.conf"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        perms         = "0600"
        uid           = "106"
      }

      template {
        data          = <<EOH
        user default on -DEBUG +@all ~* >{{with secret "secret/data/keydb/default"}}{{.Data.data.password}}{{end}}
        EOH
        destination   = "${NOMAD_SECRETS_DIR}/users.acl"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        perms         = "0600"
        uid           = "106"
      }


      volume_mount {
        volume      = "flash"
        destination = "/opt/keydb/flash"
        read_only   = false
      }

      volume_mount {
        volume      = "rdb"
        destination = "/opt/keydb/rdb"
        read_only   = false
      }
    }

    task "rotate-password" {
      driver = "exec"
      user   = "keydb"

      lifecycle {
        hook    = "poststart"
        sidecar = false
      }

      template {
        data          = <<EOH
        {
          "plugin_name": "redis-database-plugin",
          "allowed_roles": "access-keydb",
          "host": "{{ env "NOMAD_IP_redis" }}",
          "port": "{{ env "NOMAD_PORT_redis" }}",
          "username": "default",
          "password": "{{with secret "secret/data/keydb/default"}}{{.Data.data.password}}{{end}}"
        }
        EOH
        destination   = "${NOMAD_SECRETS_DIR}/keydb-database.json"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        perms         = "0400"
        uid           = "106"
      }

      template {
        data = <<EOH
        #!/bin/bash 
        vault write keydb/config/pollux @${NOMAD_SECRETS_DIR}/keydb-database.json
        vault write -force keydb/rotate-root/pollux
        EOH
        destination   = "${NOMAD_ALLOC_DIR}/vault-keydb-config.sh"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        perms         = "0555"
      }

      config {
        command = "/bin/sh"
        args = ["-c", "${NOMAD_ALLOC_DIR}/vault-keydb-config.sh"]
      }
    }

    volume "flash" {
      type      = "host"
      source    = "keydb-flash"
      read_only = false
    }
    volume "rdb" {
      type      = "host"
      source    = "keydb-rdb"
      read_only = false
    }
  }
}

keydb.conf.tpl

In the above Nomad job, a KeyDB instance is deployed as a service with root password rotation, which can also be useful for Redis since it uses the Redis database plugin. The job consists of the following components:

  1. The job is set as a service with a single group containing KeyDB and a root password rotation task.
  2. Vault integration is set up with the “nomad-keydb” policy.
  3. Networking is configured in host mode, binding the KeyDB service to the “tailscale” host network on port 6379.
  4. Update and restart settings are specified for the service.
  5. The KeyDB task uses the “exec” driver and runs as the “keydb” user. It loads the configuration file from a Nomad template based on “keydb.conf.tpl”.
  6. A template is provided for the user ACL configuration, which retrieves the default password from Vault.
  7. The KeyDB task mounts two volumes, “flash” and “rdb”, for storing data.
  8. The “rotate-password” task is a poststart hook that sets up the KeyDB configuration in Vault and rotates the root password.

This configuration should provide a solid starting point for deploying a KeyDB service in a Nomad cluster, with root password management handled by Vault.