Call to function "pathexpand" failed: filesystem function disabled

Nomad
1

Hi!
I’m new to Nomad but slowly learning. Now I’m looking at downloading a artifact from a private repository.

This is how the artifact part looks like in my job:

      artifact {
        source      = "git::git@github.myenterprise.corp:repo/infra"
        destination = "local/repo"
        options {
          sshkey = "${base64encode(file(pathexpand("~/.ssh/id_ed25519")))}"
        }
      }
      template {
        source        = "local/repo/artifacts/prometheus/prometheus.yml.tpl"
        destination = "local/prometheus.yml"
        change_mode   = "signal"
        change_signal = "SIGHUP"
      }

When I try to run this, Nomad says

Parse Error
input.hcl:26,41-52: Error in function call; Call to function “pathexpand” failed: filesystem function disabled. input.hcl:26,20-76: Unsuitable value type; Unsuitable value: value must be known

I’ve followed these steps issue#10036

  • Nomad is installed via apt (hashicorp repository)
  • Nomad is running as root
  • Nomad version: Nomad v1.1.6

I’m having trouble finding any answers what to do so I turn to the community with hopes

2

Hi @tomasaberg,

Thanks for using Nomad!

According to the documentation " The pathexpand function expands a leading ~ character to the current user’s home directory". Are you able to verify that that file exists at that path in the root users home directory? If not, do you see it in some other directory?

  • Derek and the Nomad Team
3

Hi Derek, thank you for your swift reply!

Like you say, I thought the path should be expanded to /root/.ssh/id_ed25519.
I’m able to stat the files, both with /root/ and ~/.

# stat ~/.ssh/id_ed25519
  File: /root/.ssh/id_ed25519
  Size: 419             Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 3511986     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-10-21 19:50:09.931831242 +0200
Modify: 2021-10-21 19:06:36.438124333 +0200
Change: 2021-10-21 19:06:36.438124333 +0200
 Birth: -

# stat /root/.ssh/id_ed25519
  File: /root/.ssh/id_ed25519
  Size: 419             Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 3511986     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-10-21 19:50:09.931831242 +0200
Modify: 2021-10-21 19:06:36.438124333 +0200
Change: 2021-10-21 19:06:36.438124333 +0200
 Birth: -

This is what catch my eye:
“filesystem function disabled”
Is this some configuration thing that needs to be enabled?

Is it possible to use ed25519 or do I need to use RSA keys?

Regards

4

That caught my eye too, but I found this thread which shows a fix being merged in Jan. I am not aware of any reason that would be disabled. I’ll keep digging and see if I can reproduce or there’s a regression. As for ed25519, I don’t think that’s a problem. Some searches show other people using it.

5

Cool, thank you.

Is there another way to send the base64 encoded string or is the recommended way to use
${base64encode(file(pathexpand()))?

To be a little more helpful, here is the whole job

job "prometheus" {
  datacenters = ["sth"]
  type        = "service"

  group "monitoring" {
    count = 3
    network {
      port  "prometheus" {
        to = 9090
      }
      port  "consul" {
        to = 8500
      }
    }

    task "prometheus" {
      vault {        
        policies = ["policy-kv-secrets"]
        change_mode   = "signal"
        change_signal = "SIGUSR1"
      }
      artifact {
        source      = "git::git@github.myenterprise.corp:yolo/infra"
        destination = "local/repo"
        options {
          sshkey = "${base64encode(file(pathexpand("/root/.ssh/id_ed25519")))}"
        }
      }
      template {
        source        = "${NOMAD_TASK_DIR}/repo/nomad/artifacts/prometheus/prometheus.yml.tpl"
        destination = "local/prometheus.yml"
        change_mode   = "signal"
        change_signal = "SIGHUP"
      }

      driver = "docker"

      config {
        image = "prom/prometheus"
        ports = ["prometheus"]
        volumes = [
          "local/prometheus.yml:/etc/prometheus/prometheus.yml:ro"
        ]
      }
      service {
        name = "prometheus"
        port = "prometheus"
        tags = [
          "urlprefix-/",
          "traefik.enable=true",
          "traefik.http.routers.prometheus.rule=Host(`prometheus.myenterprise.corp`)"
        ]
        check {
          name     = "Prometheus is alive"
          type     = "http"
          path     = "/-/healthy"
          interval = "10s"
          timeout  = "2s"
        } 
      }
    }
  }
}
6

So this worked for me on macOS using ed21559 and Nomad 1.1.6 and 1.1.5 :thinking:

What os are you running this on?

7

This is a Ubuntu 20.04, running on VMware 6.7

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

I had a similar job on another machine ~3 weeks ago on an Ubuntu 20.04 as well, it worked then, odd.

I guess what’s changed is the release of Ubuntu 20.04.3 :thinking:

8

Yeah that is odd, but not unheard of. Is it an option for you to rollback the ubuntu version and retest? Also, when you tested a few weeks ago were you on 1.1.6? It’s relatively new. Like I said, it worked on 1.1.6 and 1.1.5 for me, but I’m just trying to identify possible variances.

9

Sure, I’m just doing this for learing myself and setting up a poc, I’ll do whatever I can to help out.

So, this is what I’ve done:

  1. Upgraded the OS with the latest updates (noting related though, but just in case)
  2. Downgraded down to 1.1.5 → 1.1.4

Then, I come to think of, one of the differences now vs back then, is that I’m executing the jobs from the web ui. It failed when I was trying to run the job on all versions through the web ui but it worked when I used the nomad job run prometheus.hcl

Well, I’ve got it working, but I wonder why it fails in the web ui.
Should I try and upgrade back to 1.1.6?

Edit: also worth mentioning; I can’t see anything suspicious when running
journalctl -xef -u nomad (not in debug mode though)

10

I think upgrading back to 1.1.6 should be fine. Thanks for sharing the extra detail! The Web UI is now the hot path of investigation. I’ll talk to my colleagues and see what we can figure out.

11

@tomasaberg

Do you mind opening a GitHub issue for this? I think it’s more properly tracked there. If you could include a link to this discuss thread that would be awesome!

My guess is the UI handler is using HCLv1.

12

Yes sir!

I hope it’s good enough :slight_smile:

13

I don’t think you are able to read local files when submitting jobs via the web-ui (atleast I’d expect a popup regarding accessing your localdrive by your browser.)

If I remember correctly, the filesystem-functions were added along with the hcl2 support.

Functions are evaluated by the CLI during configuration parsing rather than job run time, so this function can only be used with files that are already present on disk on operator host.

If you manually base64encode the key & use the string in place of the file()-function, I’d assume it works fine?

EDIT: I see now that this was already answered in the github-issue

14

I saw this issue so I didn’t try it. It is a bit old now, but from what I understand (at the time) it wasn’t supported to just use a encoded string?

I will mark kds-runes answer as a solution for this specific issue though

15

Ok, might be true. I have struggled “a bit” with the artifact-stanca myself as well :smiley:

The specific issue you linked I believe I solved by scanning (ssh-keyscan) & adding the bitbucket/github/etc fingerprints to all of my nomad nodes “known_hosts” file.

HCL2 supports “decoding” the base64 string before submitting the job, so that might work (base64decode(“pre-encoded-b64string”))?

You should also keep in mind that the key(s) will be visible in the jobfile, and anyone with access to this file (or the job-definition in nomad) can access the key. The only way around this is using vault or similar solution.

Lately I just use a sidecar-task with “sidecar=false” and “hook=prestart” for downloading any “problematic” artifacts to /alloc/some-folder & my main task can access the files from there.

16

Sorry for late reply.
I will try out your vault suggestion. Thanks!