Simple job failed MariaDB

Hello,

nomad -v
Nomad v0.12.9 (45c139e53f2407a44b1290385b5818b46ea3a62c)
consul -v
Consul v1.8.4
Revision 12b16df32
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

I try to exec simple job mariadb:

job "mariadb" {
  region      = "global"
  datacenters = ["dc1"]
  type        = "service"

  group "database" {
    count = 1
    network {
      mode = "bridge"
      port "db" {
        to = 3306
      }
    }

    service {
      name = "mariadb"

      port = "db"

      tags = [
              "traefik.enable=true",
              "traefik.tcp.routers.mariadb.rule=HostSNI(`*`)",
              "traefik.tcp.routers.mariadb.service=mariadb",
              "traefik.tcp.services.mariadb.loadbalancer.server.port=3306",
      ]

      check {
          type     = "tcp"
          port     = "db"
          interval = "10s"
          timeout  = "2s"
      }
    }  
    task "mariadb" {
      driver = "docker"

      config {
        image        = "mariadb:10.5.8"
        network_mode = "database"

        volumes = [
          "/srv/live/mariadb/data:/var/lib/mysql",
        ]
      }
      env = {
        "MYSQL_ROOT_PASSWORD" = "password"
      }
    }
  }
}

FAILED! :’(

I have this error:

failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to initialize table forwarding rules: failed to list iptables chains: running [/usr/sbin/iptables -t filter -S --wait]: exit status 4: Fatal: can't open lock file /run/xtables.lock: Permission denied 

And in systemctl journal:

Dec 10 11:39:00 portecontainer nomad[919503]:     2020-12-10T11:39:00.095+0100 [INFO]  client.gc: marking allocation for GC: alloc_id=258600ec-5567-53ef-f63e-95c5e611092f
Dec 10 11:39:00 portecontainer nomad[919503]:  client.gc: marking allocation for GC: alloc_id=258600ec-5567-53ef-f63e-95c5e611092f
Dec 10 11:39:00 portecontainer nomad[919503]:     2020-12-10T11:39:00.095+0100 [ERROR] client.alloc_runner.runner_hook: failed to cleanup network for allocation, resources may have leaked: alloc_id=258600ec-5567-53ef-f63e-95c5e611092f alloc=258600ec-5567-53ef-f63e-95c5e611092f error="failed to find plugin "portmap" in path [/opt/cni/bin]"
Dec 10 11:39:00 portecontainer nomad[919503]: client.alloc_runner.runner_hook: failed to cleanup network for allocation, resources may have leaked: alloc_id=258600ec-5567-53ef-f63e-95c5e611092f alloc=258600ec-5567-53ef-f63e-95c5e611092f error="failed to find plugin "portmap" in path [/opt/cni/bin]"

I see in other topic to enable consul connect, that I add in consul config.json:

    "connect": {
              "enabled": true
    }

But no more works.

My job Traefik works fine.

An you help me?
Thanks

Hi @fred-gb! This bit of the error message looks to be the relevant bit:

failed to list iptables chains: running [/usr/sbin/iptables -t filter -S --wait]: exit status 4: Fatal: can't open lock file /run/xtables.lock: Permission denied 

That’s bubbling up from the CNI plugin that’s trying to create a iptable entry. Are you running the Nomad client agent as root? If so, if you were to run sudo /usr/sbin/iptables -t filter -S --wait on that machine, what do you see?

Thanks @tgross

sudo /usr/sbin/iptables -t filter -S --wait
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-a9c142acf14c -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a9c142acf14c -j DOCKER
-A FORWARD -i br-a9c142acf14c ! -o br-a9c142acf14c -j ACCEPT
-A FORWARD -i br-a9c142acf14c -o br-a9c142acf14c -j ACCEPT
-A FORWARD -o br-cee1a0e6e793 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-cee1a0e6e793 -j DOCKER
-A FORWARD -i br-cee1a0e6e793 ! -o br-cee1a0e6e793 -j ACCEPT
-A FORWARD -i br-cee1a0e6e793 -o br-cee1a0e6e793 -j ACCEPT
-A FORWARD -o br-4755b6353bcd -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-4755b6353bcd -j DOCKER
-A FORWARD -i br-4755b6353bcd ! -o br-4755b6353bcd -j ACCEPT
-A FORWARD -i br-4755b6353bcd -o br-4755b6353bcd -j ACCEPT
-A FORWARD -o br-314098d28aa9 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-314098d28aa9 -j DOCKER
-A FORWARD -i br-314098d28aa9 ! -o br-314098d28aa9 -j ACCEPT
-A FORWARD -i br-314098d28aa9 -o br-314098d28aa9 -j ACCEPT
-A FORWARD -o br-637f9b6d7598 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-637f9b6d7598 -j DOCKER
-A FORWARD -i br-637f9b6d7598 ! -o br-637f9b6d7598 -j ACCEPT
-A FORWARD -i br-637f9b6d7598 -o br-637f9b6d7598 -j ACCEPT
-A FORWARD -o br-148616416adc -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-148616416adc -j DOCKER
-A FORWARD -i br-148616416adc ! -o br-148616416adc -j ACCEPT
-A FORWARD -i br-148616416adc -o br-148616416adc -j ACCEPT
-A FORWARD -o br-ae495f9a9dd6 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-ae495f9a9dd6 -j DOCKER
-A FORWARD -i br-ae495f9a9dd6 ! -o br-ae495f9a9dd6 -j ACCEPT
-A FORWARD -i br-ae495f9a9dd6 -o br-ae495f9a9dd6 -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-ae495f9a9dd6 -o br-ae495f9a9dd6 -p tcp -m tcp --dport 9090 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-ae495f9a9dd6 ! -o br-ae495f9a9dd6 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-a9c142acf14c ! -o br-a9c142acf14c -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-637f9b6d7598 ! -o br-637f9b6d7598 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-4755b6353bcd ! -o br-4755b6353bcd -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-314098d28aa9 ! -o br-314098d28aa9 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-148616416adc ! -o br-148616416adc -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-cee1a0e6e793 ! -o br-cee1a0e6e793 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-ae495f9a9dd6 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-a9c142acf14c -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-637f9b6d7598 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-4755b6353bcd -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-314098d28aa9 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-148616416adc -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-cee1a0e6e793 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

and my systemd service:

### BEGIN INIT INFO
# Provides:          nomad
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: distributed scheduler
# Description:       distributed, highly available, datacenter-aware scheduler
### END INIT INFO

[Unit]
Description=nomad agent
Documentation=https://nomadproject.io/docs/
After=network-online.target
Wants=network-online.target

[Service]
User=ubuntu
Group=bin
ExecStart=/usr/local/bin/nomad agent -config=/etc/nomad.d

ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=42s
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity

[Install]
WantedBy=multi-user.target

user ubuntu has sudo rights.

user ubuntu has sudo rights.

Nomad doesn’t exec the command using sudo.

ok, thanks.

So, I changed to root user in systemd file. It’s little bit better. I have another error ! Yeeaaaah!

failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: failed to find plugin "bridge" in path [/opt/cni/bin]

I think this can resolve:

I will try tomorrow.

Is it secure to launch nomad with nomad user?

Thanks

Hello,

So I follow instructions and it’s work. But with other error and strange behavior.

When I add “check configuration”. Nomad job not deploying. Failed and unhealthy. It’s like if unable to check TCP without authentification.

When I delete “check configuration”. It’s works! But… Not long…

I’m unable to connect to mariadb:

mysql -u root -p -h portecontainer.lan
Enter password:
ERROR 2002 (HY000): Can't connect to MySQL server on 'portecontainer.lan' (115)

Maybe a password issue, because I need to change my password, not accepted by nomad UI with specials characters within, but volume mount is existing data with older version with older password.

Traeifk UI is unreachable after somes minutes.
Firts minutes, I can browse in UI, HTTP, TCP but after somes minutes, UI is totally unreachable. But other routers rules works.

I’m very sorry if I don’t understand something.

Thanks