Runtime not allowed

Nomad
#1

Bonjour, :wave:

Nomad: 1.2.6
OS: Ubuntu 20.04

I would like to run an VM-like docker container with runtime nestybox .

But I have this error:

Failed to create container configuration for image "xxx:latest" ("sha256:xxx"): requested runtime "sysbox-runc" is not allowed

When I ran:

docker run --runtime=sysbox-runc --rm -it --hostname my_cont registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docke

No problem

How I can allow this new runtime in nomad?

Thanks! :slightly_smiling_face:

#2

Hi @fred-gb !

Have you tried configuring allow_runtimes in the docker driver config on the Client?

#3

Thanks,

No, I didn’t find instruction in doc before you give me the link. Thanks! :slightly_smiling_face:

I added to my nomad conf this:

plugin "docker" {
        allow_runtimes = ["runc", "sysbox-runc"]
}

But not working after restart nomad :worried:

#4

@fred-gb It’s not working with the same error as before?

#5

Yes, the same.

client.hcl:

client {
    enabled = true

    node_class = ""
    no_host_uuid = false


    max_kill_timeout = "30s"

    network_speed = 0
    cpu_total_compute = 0

    gc_interval = "1m"
    gc_disk_usage_threshold = 80
    gc_inode_usage_threshold = 70
    gc_parallel_destroys = 2

    reserved {
        cpu = 0
        memory = 0
        disk = 0
    }




    options = {
        "driver.raw_exec.enable" = "true"
        "docker.privileged.enabled" = "true"
        "docker.volumes.enabled" = "true"
    }

}

plugin "docker" {
        allow_runtimes = ["runc", "sysbox-runc"]
}

base.hcl:

name = "xxx"
region = "global"
datacenter = "dc1"

enable_debug = false
disable_update_check = false


bind_addr = "0.0.0.0"
advertise {
    http = "xxx:4646"
    rpc = "xxx:4647"
    serf = "xxx:4648"
}
ports {
    http = 4646
    rpc = 4647
    serf = 4648
}


consul {
    # The address to the Consul agent.
    address = "localhost:8500"
    ssl = false
    ca_file = ""
    cert_file = ""
    key_file = ""
    token = ""
    # The service name to register the server and client with Consul.
    server_service_name = "nomad-servers"
    client_service_name = "nomad-clients"
    tags = {}

    # Enables automatically registering the services.
    auto_advertise = true

    # Enabling the server and client to bootstrap using Consul.
    server_auto_join = true
    client_auto_join = true
}

data_dir = "/var/nomad"

log_level = "INFO"
enable_syslog = true

leave_on_terminate = true
leave_on_interrupt = false


acl {
    enabled = false
    token_ttl = "30s"
    policy_ttl = "30s"
    replication_token = ""
}


vault {
    enabled = true
    address = "http://127.0.0.1:8200"
    allow_unauthenticated = true
    create_from_role = "nomad-cluster"
    task_token_ttl = ""
    ca_file = ""
    ca_path = ""
    cert_file = ""
    key_file = ""
    tls_server_name = ""
    tls_skip_verify = false
    namespace = ""
    token = "s.xxxxxx"
}


telemetry {
    disable_hostname = "false"
    collection_interval = "1s"
    use_node_name = "false"
    publish_allocation_metrics = "false"
    publish_node_metrics = "false"
    filter_default = "true"
    prefix_filter = []
    disable_dispatched_job_summary_metrics = "false"
    statsite_address = ""
    statsd_address = ""
    datadog_address = ""
    datadog_tags = []
    prometheus_metrics = "true"
    circonus_api_token = ""
    circonus_api_app = "nomad"
    circonus_api_url = "https://api.circonus.com/v2"
    circonus_submission_interval = "10s"
    circonus_submission_url = ""
    circonus_check_id = ""
    circonus_check_force_metric_activation = "false"
    circonus_check_instance_id = ""
    circonus_check_search_tag = ""
    circonus_check_display_name = ""
    circonus_check_tags = ""
    circonus_broker_id = ""
    circonus_broker_select_tag = ""
}

autopilot {
    cleanup_dead_servers      = true
    last_contact_threshold    = "200ms"
    max_trailing_logs         = 250
    server_stabilization_time = "10s"
}

and server.hcl:

server {
    enabled = true

    bootstrap_expect = 1


    rejoin_after_leave = false

    enabled_schedulers = ["service","batch","system"]
    num_schedulers = 4

    node_gc_threshold = "24h"
    eval_gc_threshold = "1h"
    job_gc_threshold = "4h"
    deployment_gc_threshold = "1h"

    encrypt = ""

    raft_protocol = 3
}

All generated by ansible-community roles.

Thanks

#6

@fred-gb can you put the allow_runtimes inside the config block in the docker plugin stanza?

e.g.

plugin "docker" {
    config {
        allow_runtimes = ["runc", "sysbox-runc"]
    }
}

TBH it’s a little surprising what you have gets passed validation :thinking:

#7

Hi,
It’s little bit better, first step passed. Bu now…

Failed to start container a8a8c3444a7bd683228c2f50204886ccdf034d0f0a0d32e3d4fe3a5dc5615315: API error (500): failed to create shim: OCI runtime create failed: container_linux.go:419: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:67: setting up rootfs mounts caused: rootfs_linux.go:1122: mounting "sysfs" to rootfs "/var/lib/sysbox/shiftfs/d52ee253-fb07-423e-b504-7c71f14c87c2" at "sys" caused: operation not permitted: unknown

and in syslog:

Apr  5 08:22:20 registry containerd[469]: time="2022-04-05T08:22:20.378637661+02:00" level=info msg="cleaning up dead shim"
Apr  5 08:22:20 registry containerd[469]: time="2022-04-05T08:22:20.426758525+02:00" level=warning msg="cleanup warnings time=\"2022-04-05T08:22:20+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=72063\ntime=\"2022-04-05T08:22:20+02:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby/a003dca36800ade5ca27cfdc17fc33c51095ecdd51d42144f477353945176cb6/init.pid: no such file or directory\"\n"
Apr  5 08:22:20 registry containerd[469]: time="2022-04-05T08:22:20.427101202+02:00" level=error msg="copy shim log" error="read /proc/self/fd/68: file already closed"
Apr  5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.428049423+02:00" level=error msg="stream copy error: reading from a closed fifo"
Apr  5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.428160159+02:00" level=error msg="stream copy error: reading from a closed fifo"
Apr  5 08:22:20 registry systemd[1]: var-lib-docker-overlay2-11e0b56e38843a22deb5c9c5ec1451ae0f4bdb279cc502f356f8f66810b5d910-merged.mount: Succeeded.
Apr  5 08:22:20 registry systemd[68659]: var-lib-docker-overlay2-11e0b56e38843a22deb5c9c5ec1451ae0f4bdb279cc502f356f8f66810b5d910-merged.mount: Succeeded.
Apr  5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.481542295+02:00" level=error msg="a003dca36800ade5ca27cfdc17fc33c51095ecdd51d42144f477353945176cb6 cleanup: failed to delete container from containerd: no such container"
Apr  5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.481602016+02:00" level=error msg="Handler for POST /containers/a003dca36800ade5ca27cfdc17fc33c51095ecdd51d42144f477353945176cb6/start returned error: failed to create shim: OCI runtime create failed: container_linux.go:419: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:67: setting up rootfs mounts caused: rootfs_linux.go:1122: mounting \"sysfs\" to rootfs \"/var/lib/sysbox/shiftfs/6ce972b3-feae-4400-b30d-0c2d1500387a\" at \"sys\" caused: operation not permitted: unknown"

I don’t understand, but I see “sys” operation not permitted, something else to add to nomad config?

Thanks

#8

I think it’s related to this:

But I don’t know which allow_caps to add, lot of sys, but not sysfs. If it’s the right way…