Nomad job - google cloud storage artifact

Nomad
1

Hi everyone,

We have a nomad task that is supposed to start a java application and artifact block to download the jar file from a private google cloud storage.
The problem we face is that the authentication to Google Cloud Storage fails with
“failed to download artifact “gcs::https://www.googleapis.com/storage/v1/our-bucket/our-file.jar”: googleapi: Error 403: Insufficient Permission, insufficientPermissions”

We tried using:
GOOGLE_CREDENTIALS using stringified credentials string
GOOGLE_APPLICATION_CREDENTIALS point to the service account json file available to the task
GOOGLE_OAUTH_TOKEN

None of above worked showing 403 insufficient permissions.

This is our current config:
task “app” {

  driver = "java"
  config {
    jar_path    = "local/our-file.jar"
    jvm_options = ["-Xmx2048m", "-Xms256m"]
  }
  env {
    GOOGLE_APPLICATION_CREDENTIALS = "/etc/creds.json"
  }
  artifact {
    source = "gcs::https://www.googleapis.com/storage/v1/our-bucket/our-file.jar"
  }
}

What is more interesting is that we downloaded the go-getter module, and ran it locally - it did work given that either GOOGLE_APPLICATION_CREDENTIALS or GOOGLE_OAUTH_TOKEN were present as env variables.

Does anyone have experience using artifact block pointing to non-public Google Cloud Storage bucket?

Thank you!

1 Like
2

Hi @ivan - for private GCS our go-getter library is a bit less convenient to use than private S3, etc.

The basic problem is the environment variables need to be set on the Nomad Client agent - not on the job spec. If you’re running Nomad under systemd, you’d add them to the unit file, e.g.

[Service]
Environment="GOOGLE_OAUTH_ACCESS_TOKEN=xxxx"

Starting with Nomad 1.5, you’ll also need to allow the artifact download sandbox to have access to this environment variable by setting artifact.set_environment_variables in Client configuration - nomad/upgrade-specific.mdx at v1.5.0-rc.1 · hashicorp/nomad · GitHub

3

Hi Seth,

Many thanks for your clarification!
We followed the steps using GOOGLE_APPLICATION_CREDENTIALS (since the OAUTH Token is subject to expiration). We’re using Nomad 1.4.4.

  1. In client.hcl:
    client {
    enabled = true
    node_class = “hashistack”

options {
“driver.raw_exec.enable” = “1”
“docker.privileged.enabled” = “true”
“env.denylist” = “CUSTOM_VAR”
}
artifact{
“set_environment_variables” = “GOOGLE_APPLICATION_CREDENTIALS”
}
}
2) In systemd service definition
[Service]
Restart=on-failure
ExecStart=/usr/local/bin/nomad agent -config=“/etc/nomad.d/nomad.hcl”
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
User=root
Group=root
ENVIRONMENT=“GOOGLE_APPLICATION_CREDENTIALS=/etc/creds.json”

  1. we restarted the nomad service
  2. we ran our job - same problem:
    “failed to download artifact “gcs::https://www.googleapis.com/storage/v1/ourbucket/jar_files%2Ftesting-junit5-mockito-1.0-447e220.jar”: googleapi: Error 403: Insufficient Permission, insufficientPermissions”

image
image1398×103 10.3 KB

Can you please advise/or if you have a working end-to-end example:

  1. how can we use GOOGLE_APPLICATION_CREDENTIALS
  2. if you plan to introduce the same integration capabilities for Cloud Storage like AWS S3 (EC2 IAM Instance profiles) - so that go-getter library can download the GCS artifact using the Compute Engine Service account directly?

Reason I ask - is because having credentials stored directly on the host is not a recommended security practice by Google.

Thank you in advance!