# Runtime not allowed

**URL:** <https://discuss.hashicorp.com/t/runtime-not-allowed/37960>\
**Category:** Nomad\
**Created:** [April 4, 2022, 1:06pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960 "2022-04-04T13:06:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fred-gb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/fred-gb/32/10243_2.png) [@fred-gb](https://discuss.hashicorp.com/u/fred-gb)\
**Post date:** [April 4, 2022, 1:06pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/1 "2022-04-04T13:06:51Z")

</div>

Bonjour, 👋

Nomad: 1.2.6  
OS: Ubuntu 20.04

I would like to run an VM-like docker container with runtime [nestybox](https://github.com/nestybox/sysbox) .

But I have this error:

```auto
Failed to create container configuration for image "xxx:latest" ("sha256:xxx"): requested runtime "sysbox-runc" is not allowed

```

When I ran:

```auto
docker run --runtime=sysbox-runc --rm -it --hostname my_cont registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docke

```

No problem

How I can allow this new runtime in nomad?

Thanks! 🙂

---

<div class="post-metadata">

**Author:** ![seth.hoenig](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/seth.hoenig/32/27213_2.png) [@seth.hoenig](https://discuss.hashicorp.com/u/seth.hoenig)\
**Post date:** [April 4, 2022, 3:49pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/2 "2022-04-04T15:49:20Z")

</div>

Hi @fred-gb !

Have you tried configuring [allow\_runtimes](https://www.nomadproject.io/docs/drivers/docker#allow_runtimes) in the docker driver config on the Client?

---

<div class="post-metadata">

**Author:** ![fred-gb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/fred-gb/32/10243_2.png) [@fred-gb](https://discuss.hashicorp.com/u/fred-gb)\
**Post date:** [April 4, 2022, 4:16pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/3 "2022-04-04T16:16:26Z")

</div>

Thanks,

No, I didn’t find instruction in doc before you give me the link. Thanks! 🙂

I added to my nomad conf this:

```auto
plugin "docker" {
        allow_runtimes = ["runc", "sysbox-runc"]
}

```

But not working after restart nomad 😟

---

<div class="post-metadata">

**Author:** ![seth.hoenig](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/seth.hoenig/32/27213_2.png) [@seth.hoenig](https://discuss.hashicorp.com/u/seth.hoenig)\
**Post date:** [April 4, 2022, 4:25pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/4 "2022-04-04T16:25:38Z")

</div>

@fred-gb It’s not working with the same error as before?

---

<div class="post-metadata">

**Author:** ![fred-gb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/fred-gb/32/10243_2.png) [@fred-gb](https://discuss.hashicorp.com/u/fred-gb)\
**Post date:** [April 4, 2022, 5:13pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/5 "2022-04-04T17:13:10Z")

</div>

Yes, the same.

client.hcl:

```auto
client {
    enabled = true

    node_class = ""
    no_host_uuid = false

    max_kill_timeout = "30s"

    network_speed = 0
    cpu_total_compute = 0

    gc_interval = "1m"
    gc_disk_usage_threshold = 80
    gc_inode_usage_threshold = 70
    gc_parallel_destroys = 2

    reserved {
        cpu = 0
        memory = 0
        disk = 0
    }

    options = {
        "driver.raw_exec.enable" = "true"
        "docker.privileged.enabled" = "true"
        "docker.volumes.enabled" = "true"
    }

}

plugin "docker" {
        allow_runtimes = ["runc", "sysbox-runc"]
}

```

base.hcl:

```auto
name = "xxx"
region = "global"
datacenter = "dc1"

enable_debug = false
disable_update_check = false

bind_addr = "0.0.0.0"
advertise {
    http = "xxx:4646"
    rpc = "xxx:4647"
    serf = "xxx:4648"
}
ports {
    http = 4646
    rpc = 4647
    serf = 4648
}

consul {
    # The address to the Consul agent.
    address = "localhost:8500"
    ssl = false
    ca_file = ""
    cert_file = ""
    key_file = ""
    token = ""
    # The service name to register the server and client with Consul.
    server_service_name = "nomad-servers"
    client_service_name = "nomad-clients"
    tags = {}

    # Enables automatically registering the services.
    auto_advertise = true

    # Enabling the server and client to bootstrap using Consul.
    server_auto_join = true
    client_auto_join = true
}

data_dir = "/var/nomad"

log_level = "INFO"
enable_syslog = true

leave_on_terminate = true
leave_on_interrupt = false

acl {
    enabled = false
    token_ttl = "30s"
    policy_ttl = "30s"
    replication_token = ""
}

vault {
    enabled = true
    address = "http://127.0.0.1:8200"
    allow_unauthenticated = true
    create_from_role = "nomad-cluster"
    task_token_ttl = ""
    ca_file = ""
    ca_path = ""
    cert_file = ""
    key_file = ""
    tls_server_name = ""
    tls_skip_verify = false
    namespace = ""
    token = "s.xxxxxx"
}

telemetry {
    disable_hostname = "false"
    collection_interval = "1s"
    use_node_name = "false"
    publish_allocation_metrics = "false"
    publish_node_metrics = "false"
    filter_default = "true"
    prefix_filter = []
    disable_dispatched_job_summary_metrics = "false"
    statsite_address = ""
    statsd_address = ""
    datadog_address = ""
    datadog_tags = []
    prometheus_metrics = "true"
    circonus_api_token = ""
    circonus_api_app = "nomad"
    circonus_api_url = "https://api.circonus.com/v2"
    circonus_submission_interval = "10s"
    circonus_submission_url = ""
    circonus_check_id = ""
    circonus_check_force_metric_activation = "false"
    circonus_check_instance_id = ""
    circonus_check_search_tag = ""
    circonus_check_display_name = ""
    circonus_check_tags = ""
    circonus_broker_id = ""
    circonus_broker_select_tag = ""
}

autopilot {
    cleanup_dead_servers = true
    last_contact_threshold = "200ms"
    max_trailing_logs = 250
    server_stabilization_time = "10s"
}

```

and server.hcl:

```auto
server {
    enabled = true

    bootstrap_expect = 1

    rejoin_after_leave = false

    enabled_schedulers = ["service","batch","system"]
    num_schedulers = 4

    node_gc_threshold = "24h"
    eval_gc_threshold = "1h"
    job_gc_threshold = "4h"
    deployment_gc_threshold = "1h"

    encrypt = ""

    raft_protocol = 3
}

```

All generated by ansible-community roles.

Thanks

---

<div class="post-metadata">

**Author:** ![seth.hoenig](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/seth.hoenig/32/27213_2.png) [@seth.hoenig](https://discuss.hashicorp.com/u/seth.hoenig)\
**Post date:** [April 4, 2022, 6:34pm UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/6 "2022-04-04T18:34:02Z")

</div>

@fred-gb can you put the `allow_runtimes` inside the `config` block in the docker plugin stanza?

e.g.

```nohighlight
plugin "docker" {
    config {
        allow_runtimes = ["runc", "sysbox-runc"]
    }
}

```

TBH it’s a little surprising what you have gets passed validation 🤔

---

<div class="post-metadata">

**Author:** ![fred-gb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/fred-gb/32/10243_2.png) [@fred-gb](https://discuss.hashicorp.com/u/fred-gb)\
**Post date:** [April 5, 2022, 6:24am UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/7 "2022-04-05T06:24:50Z")

</div>

Hi,  
It’s little bit better, first step passed. Bu now…

```auto
Failed to start container a8a8c3444a7bd683228c2f50204886ccdf034d0f0a0d32e3d4fe3a5dc5615315: API error (500): failed to create shim: OCI runtime create failed: container_linux.go:419: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:67: setting up rootfs mounts caused: rootfs_linux.go:1122: mounting "sysfs" to rootfs "/var/lib/sysbox/shiftfs/d52ee253-fb07-423e-b504-7c71f14c87c2" at "sys" caused: operation not permitted: unknown

```

and in syslog:

```auto
Apr 5 08:22:20 registry containerd[469]: time="2022-04-05T08:22:20.378637661+02:00" level=info msg="cleaning up dead shim"
Apr 5 08:22:20 registry containerd[469]: time="2022-04-05T08:22:20.426758525+02:00" level=warning msg="cleanup warnings time=\"2022-04-05T08:22:20+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=72063\ntime=\"2022-04-05T08:22:20+02:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby/a003dca36800ade5ca27cfdc17fc33c51095ecdd51d42144f477353945176cb6/init.pid: no such file or directory\"\n"
Apr 5 08:22:20 registry containerd[469]: time="2022-04-05T08:22:20.427101202+02:00" level=error msg="copy shim log" error="read /proc/self/fd/68: file already closed"
Apr 5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.428049423+02:00" level=error msg="stream copy error: reading from a closed fifo"
Apr 5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.428160159+02:00" level=error msg="stream copy error: reading from a closed fifo"
Apr 5 08:22:20 registry systemd[1]: var-lib-docker-overlay2-11e0b56e38843a22deb5c9c5ec1451ae0f4bdb279cc502f356f8f66810b5d910-merged.mount: Succeeded.
Apr 5 08:22:20 registry systemd[68659]: var-lib-docker-overlay2-11e0b56e38843a22deb5c9c5ec1451ae0f4bdb279cc502f356f8f66810b5d910-merged.mount: Succeeded.
Apr 5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.481542295+02:00" level=error msg="a003dca36800ade5ca27cfdc17fc33c51095ecdd51d42144f477353945176cb6 cleanup: failed to delete container from containerd: no such container"
Apr 5 08:22:20 registry dockerd[674]: time="2022-04-05T08:22:20.481602016+02:00" level=error msg="Handler for POST /containers/a003dca36800ade5ca27cfdc17fc33c51095ecdd51d42144f477353945176cb6/start returned error: failed to create shim: OCI runtime create failed: container_linux.go:419: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:67: setting up rootfs mounts caused: rootfs_linux.go:1122: mounting \"sysfs\" to rootfs \"/var/lib/sysbox/shiftfs/6ce972b3-feae-4400-b30d-0c2d1500387a\" at \"sys\" caused: operation not permitted: unknown"

```

I don’t understand, but I see “sys” operation not permitted, something else to add to nomad config?

Thanks

---

<div class="post-metadata">

**Author:** ![fred-gb](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/fred-gb/32/10243_2.png) [@fred-gb](https://discuss.hashicorp.com/u/fred-gb)\
**Post date:** [April 5, 2022, 6:40am UTC](https://discuss.hashicorp.com/t/runtime-not-allowed/37960/8 "2022-04-05T06:40:01Z")

</div>

I think it’s related to this:

> **[Unable to mount /sys inside the container with rw access in unprivileged mode](https://forums.docker.com/t/unable-to-mount-sys-inside-the-container-with-rw-access-in-unprivileged-mode/97043)**
>
> I would need to mount /sys inside the container with rw access, in the privileged mode it is by default mounting with rw access but with unprivileged mode it is mounting with ro access. any idea how to get /sys mounted with rw access on the...

But I don’t know which allow\_caps to add, lot of sys, but not sysfs. If it’s the right way…