Packer data: NoCredentialProviders: no valid providers in chain. Deprecated

brucellino1 · September 6, 2021, 5:12pm

Hi there!

I have been testing a Packer template with the HCL2 syntax for the past few days and have finally committed it to our repo. Although it was building fine manually (on my machine), I am getting the errors below in CI:

Error: Datasource.Execute failed: no valid credential sources for  found.

Full log

[2021-09-06T15:46:16.168Z] + packer init packer.pkr.hcl
[2021-09-06T15:46:16.434Z] + packer validate -var test=false -var version=latest packer.pkr.hcl
[2021-09-06T15:46:18.364Z] + packer inspect packer.pkr.hcl
[2021-09-06T15:46:18.627Z] Packer Inspect: HCL2 mode
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z] > input-variables:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z] var.ansible_host: "focal"
[2021-09-06T15:46:18.627Z] var.region: "eu-central-1"
[2021-09-06T15:46:18.627Z] var.test: "true"
[2021-09-06T15:46:18.627Z] var.version: "next"
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z] > local-variables:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z] local.aws_access_key: "xxx:
[2021-09-06T15:46:18.627Z] local.aws_secret_key: "xxx"
[2021-09-06T15:46:18.627Z] local.github_token: "xxx"
[2021-09-06T15:46:18.627Z] local.quay_password: "xxx"
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z] > builds:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]   > focal-ami:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]     sources:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       amazon-ebs.focal
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]     provisioners:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       ansible
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]     post-processors:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       <no post-processor>
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]   > docker:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]     sources:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       docker.focal
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]     provisioners:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       ansible
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]     post-processors:
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       0:
[2021-09-06T15:46:18.627Z]         docker-tag
[2021-09-06T15:46:18.627Z]         docker-push
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.627Z]       1:
[2021-09-06T15:46:18.627Z]         docker-tag
[2021-09-06T15:46:18.627Z]         docker-push
[2021-09-06T15:46:18.627Z] 
[2021-09-06T15:46:18.891Z] + packer build -var test=false -var version=latest packer.pkr.hcl
[2021-09-06T15:46:19.156Z] Error: Datasource.Execute failed: no valid credential sources for  found.
[2021-09-06T15:46:19.156Z] 
[2021-09-06T15:46:19.156Z] Please see 
[2021-09-06T15:46:19.156Z] for more information about providing credentials.
[2021-09-06T15:46:19.156Z] 
[2021-09-06T15:46:19.156Z] Error: NoCredentialProviders: no valid providers in chain. Deprecated.
[2021-09-06T15:46:19.157Z] 	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[2021-09-06T15:46:19.157Z] 
[2021-09-06T15:46:19.157Z] 
[2021-09-06T15:46:19.157Z]   on packer.pkr.hcl line 32:
[2021-09-06T15:46:19.157Z]   (source code not available)

Some things to note:

The template (shown below) uses credentials from a Vault AWS secret, retrieved with an AppRole token. The AWS credentials are cached in a json file and read by the packer template using the file and jsondecode functions.
I have checked extensively that these credentials are valid.
1. packer inspect shows that they are valid
2. If i set them in the environment, the problem disappears
Packer init and packer validate are showing correct output in CI.
I have used this template on my development machine a lot - it definitely works.
I am using the same Vault in dev as in CI.
The CI is run with Jenkins, using an EC2 provisioner. Jenkins injects the AppRole role ID and secret ID into the environment of an EC2 instance with the same tools as my dev machine.
The agent runs in the same AWS region as the AMI I want to build.

The only thing I can think of from the error is that packer is getting AWS credentials from something on the host, according to the credentials setting order.

However, aws sts get-caller-identity shows that there are no configured credentials.

I haven’t found anything on the Github issues for packer either. What gives?

The template is as follows:

/* originally commented out
packer {
  required_plugins {
    amazon = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/amazon"
    }
    docker = {
      version = ">= 1.0.1"
      source  = "github.com/hashicorp/docker"
    }
    ansible = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/ansible"
    }
  }
}
*/
locals {
  quay_password = vault("path/to/secret "secret")
  github_token = vault("path/to/secret", "secret")
  aws_access_key = jsondecode(file("aws.json")).data.access_key
  aws_secret_key = jsondecode(file("aws.json")).data.secret_key
}

variable "test" {
  description = "Boolean toggle to skip ami creation. If set to true, AMI will _NOT_ be created."
  type    = bool
  default = true
}

variable "ansible_host" {
  description = "Hostname assigned to the focal docker instance created by the docker build."
  type = string
  default = "focal"
}

variable "region" {
  description = "AWS region to use."
  type    = string
  default = "eu-central-1"
}

variable "version" {
  description = "Semantic version used to tag the images."
  type    = string
  default = "next"
}

data "amazon-ami" "focal" {
  filters = {
    virtualization-type = "hvm"
    name                = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
    root-device-type    = "ebs"
  }
  owners      = ["099720109477"]
  most_recent = true
}

source "docker" "focal" {
  image   = "public.ecr.aws/lts/ubuntu:focal"
  commit  = true
  exec_user = "root"
  changes = [
    "ENTRYPOINT [\"jenkins-agent\"]",
    "WORKDIR /home/jenkins",
    "USER jenkins",
    "LABEL version=${var.version}"
  ]
  run_command = ["-d", "-i", "-t", "--name", "${var.ansible_host}", "{{.Image}}", "/bin/bash"]
}

source "amazon-ebs" "focal" {
  source_ami      = data.amazon-ami.focal.id
  region          = var.region
  skip_create_ami = var.test
  ami_name        = "ansible-test-harness-focal-${formatdate("YYYY-MM-DD-'T'hh-mm-ssZ", timestamp())}"
  instance_type   = "t3.micro"
  ssh_interface   = "private_ip"
  ssh_username    = "ubuntu"
  access_key = local.aws_access_key
  secret_key = local.aws_secret_key
  force_deregister = true
  force_delete_snapshot = true
  associate_public_ip_address = false
  shutdown_behavior = "terminate"
  launch_block_device_mappings {
    device_name = "/dev/sda1"
    volume_size = 40
    volume_type = "gp2"
    delete_on_termination = true
  }
  run_tags = {
    Name = "packer-builder-test-harness"
    stateful = false
    application = "jenkins"
    tier = "agent"
  }
  subnet_filter {
    filters = {
     // other filters
      "availability-zone" = "eu-central-1a"
    }
    most_free = true
  }
  security_group_filter {
    filters = {
      "xxx" : "xxx"
    }
  }
}

build {
  name    = "focal-ami"
  sources = ["source.amazon-ebs.focal"]

  provisioner "ansible" {
    only          = ["amazon-ebs.focal"]
    groups        = ["ec2"]
    playbook_file = "playbook.yml"
    user          = "ubuntu"
    ansible_env_vars = [
      "ANSIBLE_ROLES_PATH=$PWD/../../",
      "ANSIBLE_STDOUT_CALLBACK=yaml",
      "ANSIBLE_REMOTE_USER=ubuntu"
    ]
  }
}


build {
  name = "docker"
  sources = ["source.docker.focal"]

  provisioner "ansible" {
    only          = ["docker.focal"]
    groups        = ["docker"]
    playbook_file = "playbook.yml"
    user          = "root"
    ansible_env_vars = [
      "ANSIBLE_ROLES_PATH=$PWD/../../",
      "ANSIBLE_LOAD_CALLBACK_PLUGINS=1",
      "ANSIBLE_CALLBACK_WHITELIST=yaml,profile_tasks",
      # "ANSIBLE_REMOTE_USER=root",
      "ANSIBLE_PYTHON_INTERPRETER=python3",
      "ANSIBLE_TRANSPORT=docker"
    ]
    extra_arguments = [
      "-e ansible_host=${var.ansible_host}",
      "-e ansible_connection=docker",
      "-e",  "ansible_user=root"
    ]
  }

  post-processors {
    // redacted
}

update 09.09.2021 : seems to be a similar issue to

update 09.09.2021 : Seems that Packer data sources want the AWS credential chain, but don’t accept access_key and secret_key.

brucellino1 · September 9, 2021, 11:17am

Ok, I think I have found out what’s going on.

I’m using locals to set AWS creds, but locals are evaluated after data sources according to

github.com/hashicorp/packer

Data Sources cannot use local values

opened 05:25PM - 11 May 21 UTC

KOConchobhair

bug hcl2 hcl2-dag

#### Overview of the Issue Trying to use local values in a data source config…uration and getting errors from packer validate. #### Reproduction Steps ``` packer init foo.pkr.hcl packer validate foo.pkr.hcl ``` ### Packer version `Packer v1.7.2` ### Simplified Packer Buildfile ``` packer { required_version = ">= 1.7.0" required_plugins { amazon = { version = ">= 0.0.1" source = "github.com/hashicorp/amazon" } } } locals { assume_role_arn = "arn:aws:iam::1234567890:role/allow-access-from-other-accounts" assume_role_session = "packer" assume_role_duration = 3600 } data "amazon-ami" "ubuntu_ami" { assume_role { role_arn = local.assume_role_arn session_name = local.assume_role_session duration_seconds = local.assume_role_duration } filters = { name = "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["099720109477"] # This is the ID for Canonical region = "us-east-1" } ``` ### Operating system and Environment details Linux KOCONNOR 4.4.0-17763-Microsoft #1432-Microsoft Mon Aug 18 18:18:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux (WSL in Windows 10) ### Log Fragments and crash.log files ``` 2021/05/11 10:53:12 [INFO] Packer version: 1.7.2 [go1.16.3 linux amd64] 2021/05/11 10:53:12 [TRACE] discovering plugins in /usr/local/bin 2021/05/11 10:53:12 [TRACE] discovering plugins in /home/koconnor/.packer.d/plugins 2021/05/11 10:53:12 [TRACE] discovering plugins in . 2021/05/11 10:53:12 [INFO] PACKER_CONFIG env var not set; checking the default config file path 2021/05/11 10:53:12 [INFO] PACKER_CONFIG env var set; attempting to open config file: /home/koconnor/.packerconfig 2021/05/11 10:53:12 [WARN] Config file doesn't exist: /home/koconnor/.packerconfig 2021/05/11 10:53:12 [INFO] Setting cache directory: /mnt/c/Users/koconnor/Work/git/princetonidentity/identity-server-cloud/identity-server/packer_cache e: Running in background, not using a TTY 2021/05/11 10:53:12 [TRACE] listing potential installations for "github.com/hashicorp/amazon" that match ">= 0.0.1". plugingetter.ListInstallationsOptions{FromFolders:[]string{"/usr/local/bin/packer", ".", "/home/koconnor/.packer.d/plugins"}, BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"linux", ARCH:"amd64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0xc000296880)}}}} 2021/05/11 10:53:12 [TRACE] Found the following "github.com/hashicorp/amazon" installations: [{/home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 v0.0.1}] 2021/05/11 10:53:12 [INFO] found external [chroot ebs ebssurrogate ebsvolume instance] builders from amazon plugin 2021/05/11 10:53:12 [INFO] found external [import] post-processors from amazon plugin 2021/05/11 10:53:12 found external [ami secretsmanager] datasource from amazon plugin 2021/05/11 10:53:12 [TRACE] Starting external plugin /home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 start datasource ami 2021/05/11 10:53:12 Starting plugin: /home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 []string{"/home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64", "start", "datasource", "ami"} 2021/05/11 10:53:12 Waiting for RPC address for: /home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 2021/05/11 10:53:12 packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 plugin: 2021/05/11 10:53:12 Plugin address: unix /tmp/packer-plugin458151726 2021/05/11 10:53:12 packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 plugin: 2021/05/11 10:53:12 Waiting for connection... 2021/05/11 10:53:12 Received unix RPC address for /home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64: addr is /tmp/packer-plugin458151726 2021/05/11 10:53:12 packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 plugin: 2021/05/11 10:53:12 Serving a plugin connection... 2021/05/11 10:53:12 packer-plugin-amazon_v0.0.1_x5.0_linux_amd64 plugin: 2021/05/11 10:53:12 [TRACE] starting datasource ami Error: Unsupported attribute on foo.pkr.hcl line 20: (source code not available) on foo.pkr.hcl line 20: (source code not available) This object does not have an attribute named "assume_role_arn". This object does not have an attribute named "assume_role_arn". Error: Unsupported attribute Error: Unsupported attribute on foo.pkr.hcl line 22: (source code not available) on foo.pkr.hcl line 22: (source code not available) This object does not have an attribute named "assume_role_duration". This object does not have an attribute named "assume_role_duration". Error: Unsupported attribute Error: Unsupported attribute on foo.pkr.hcl line 21: (source code not available) on foo.pkr.hcl line 21: (source code not available) This object does not have an attribute named "assume_role_session". 2021/05/11 10:53:12 [INFO] (telemetry) Finalizing. This object does not have an attribute named "assume_role_session". 2021/05/11 10:53:12 waiting for all plugin processes to complete... 2021/05/11 10:53:12 /home/koconnor/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v0.0.1_x5.0_linux_amd64: plugin process exited ```

I guess the way to do this is to use the typical AWS credential chain, until the locals are included in the DAG.

Hope I’m getting this right, happy for some critical insight

SwampDragons · September 9, 2021, 6:42pm

That’s correct – you can’t evaluate locals inside data sources. The DAG is something that’ll probably be a bit in the future still, so I’m afraid you’ll need to find another workaround for now.