Packer source blocks unable to validate in packer 1.11.2

Packer
1

Following the docs,
I’m trying to validate my packer template after updating to 1.11.2 to use the source blocks, but something is wrong and its not clear to me what. I couldn’t figure out from the documentation how we would make sources (target images) dynamic. In this example, depending on the image we have to change the user we connect as to perform other operations, like create a new user.

This is a reduced example which targets different source images with different users to make a new user using ansible.

source "users" "newuser" {
  name = "create_users"

  syscontrol_gid = local.syscontrol_gid

  provisioner "ansible" { # Add user deployuser
    playbook_file = "./ansible/newuser.yaml"
    user          = "rocky"
    extra_arguments = [
      "-v",
      "--extra-vars",
      "variable_user=${source.variable_user} sudo=true passwordless_sudo=true add_to_group_syscontrol=${source.add_to_group_syscontrol} variable_connect_as_user=${source.variable_connect_as_user} variable_uid=${source.variable_uid} syscontrol_gid=${source.syscontrol_gid} variable_host=default delegate_host=localhost"
    ]
    collections_path = "./ansible/collections"
    roles_path       = "./ansible/roles"
    ansible_env_vars = ["ANSIBLE_CONFIG=ansible/ansible.cfg"]
    galaxy_file      = "./requirements.yml"
    only = source.only
  }
}

build {

  source "users.newuser" {
    variable_user = "deployuser"
    variable_uid = local.deployuser_uid
    add_to_group_syscontrol = true
    variable_connect_as_user = "rocky"
    only = [
      "amazon-ebs.rocky8-rendernode-ami",
    ]
  }

  source "users.newuser" {
    variable_user = "deployuser"
    variable_uid = local.deployuser_uid
    add_to_group_syscontrol = true
    variable_connect_as_user = "ec2-user"
    only = [
      "amazon-ebs.amznlnx2023-rendernode-ami",
    ]
  }
}

These are the errors I get from packer validate, it seems to repeat itself unnecessarily as well.


Error: missing source reference

  on ./firehawk-ami.pkr.hcl line 905:
  (source code not available)

a build block must reference at least one source to be built

Error: Unknown source type users

  on ./firehawk-ami.pkr.hcl line 852:
  (source code not available)

The source users is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.

https://developer.hashicorp.com/packer/integrations?filter=users

Error: Unknown source type users

  on ./firehawk-ami.pkr.hcl line 852:
  (source code not available)

The source users is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.

https://developer.hashicorp.com/packer/integrations?filter=users

Error: Unknown source type users

  on ./firehawk-ami.pkr.hcl line 852:
  (source code not available)

The source users is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.

https://developer.hashicorp.com/packer/integrations?filter=users

Error: Unknown source type users

  on ./firehawk-ami.pkr.hcl line 852:
  (source code not available)

The source users is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.

https://developer.hashicorp.com/packer/integrations?filter=users

Error: Unknown source type users

  on ./firehawk-ami.pkr.hcl line 852:
  (source code not available)

The source users is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.

https://developer.hashicorp.com/packer/integrations?filter=users

I’m also finding the HCL plugin for vscode is not linting correctly and errors for ever build bock you attempt to create.

2

Hi @queglay,

There are multiple things that catch my eye in there, so please bear with me.

Regarding your source definition, they shouldn’t contain provisioner blocks, these can only be part of a build block so that would be one first error here.
As for your only definition, it is not supported as part of a source iirc, you can only have those as part of provisioner and post-processor, so you can have one build block with those provisioners, and they can selectively execute based on the builder being run.

The error message you get is the type of source not existing, so this is a matter of which builder you want to use for running this build, I don’t know which plugin you expect to use, I don’t know if there’s a plugin called users, but it could be a community plugin that I don’t know about.

The message you get is indeed duplicated a few times, I assume because each source block redefined is referencing the same base source, which is the cause of the problem. Not sure how we can trivially fix this one, but there might be something we can do to help.

3

I also tried something more basic and tried naming them with null

source "null" "newuser" {
  name = "create_users"
  syscontrol_gid = local.syscontrol_gid

  provisioner "ansible" { # Add user deployuser
    playbook_file = "./ansible/newuser.yaml"
    user          = "rocky"
    extra_arguments = [
      "-v",
      "--extra-vars",
      "variable_user=${source.variable_user} sudo=true passwordless_sudo=true add_to_group_syscontrol=${source.add_to_group_syscontrol} variable_connect_as_user=${source.variable_connect_as_user} variable_uid=${source.variable_uid} syscontrol_gid=${source.syscontrol_gid} variable_host=default delegate_host=localhost"
    ]
    collections_path = "./ansible/collections"
    roles_path       = "./ansible/roles"
    ansible_env_vars = ["ANSIBLE_CONFIG=ansible/ansible.cfg"]
    galaxy_file      = "./requirements.yml"
    only = source.only
  }
}

build {
  source "source.null.newuser" {
    variable_user = "deployuser"
    variable_uid = local.deployuser_uid
    add_to_group_syscontrol = true
    variable_connect_as_user = "rocky"
    only = [
      "amazon-ebs.rocky8-rendernode-ami",
    ]
  }
}

Thanks for your reply @lbajolet-hashicorp ,
Is it not possible to reuse a provisioner block or ansible plugin at all with this feature? The only plugin I’m hoping to reuse here is Ansible. I have a lot of duplicate usage of this in my templates so thats the end goal I’m hoping might be possible.

4

I will say that depends on what you want :slight_smile:

If you have the quasi-exact same provisioner used for a bunch of different builds, you can probably play with dynamic provisioner, so that in your build block you can have a provisioner definition that you can change values for depending on a variable/local/other.
This would allow you to have effectively one build block, with one provisioner definition, which would dynamically unroll to a list of provisioners.

It could be tricky to pull off to be honest, and not everything might be possible, but looking at your sample and what you’re trying to do, this might actually work.

Feel free to ping again if you need more help!

1 Like
5

I managed to get it to work. The documentation says these blocks are essentially merged, and it is possible to define only as a variable. This was my implementation in the end:


source "null" "newuser" {
  name = "create_users"
  syscontrol_gid = local.syscontrol_gid

  provisioner "ansible" { # Add user deployuser
    playbook_file = "./ansible/newuser.yaml"
    user          = "rocky"
    extra_arguments = [
      "-v",
      "--extra-vars",
      "variable_user=${source.variable_user} sudo=true passwordless_sudo=true add_to_group_syscontrol=${source.add_to_group_syscontrol} variable_connect_as_user=${source.variable_connect_as_user} variable_uid=${source.variable_uid} syscontrol_gid=${source.syscontrol_gid} variable_host=default delegate_host=localhost"
    ]
    collections_path = "./ansible/collections"
    roles_path       = "./ansible/roles"
    ansible_env_vars = ["ANSIBLE_CONFIG=ansible/ansible.cfg"]
    galaxy_file      = "./requirements.yml"
    only = source.only
  }
}

build {
  source "source.null.newuser" {
    variable_user = "deployuser"
    variable_uid = local.deployuser_uid
    add_to_group_syscontrol = true
    variable_connect_as_user = "rocky"
    only = [
      "amazon-ebs.rocky8-rendernode-ami",
    ]
  }

  source "source.null.newuser" {
    variable_user = "deployuser"
    variable_uid = local.deployuser_uid
    add_to_group_syscontrol = true
    variable_connect_as_user = "ec2-user"
    only = [
      "amazon-ebs.amznlnx2023-rendernode-ami",
    ]
  }

  source "source.null.newuser" {
    variable_user = "deadlineuser"
    variable_uid = local.deadlineuser_uid
    add_to_group_syscontrol = false
    variable_connect_as_user = "rocky"
    only = [
      "amazon-ebs.rocky8-rendernode-ami",
    ]
  }

  source "source.null.newuser" {
    variable_user = "deadlineuser"
    variable_uid = local.deadlineuser_uid
    add_to_group_syscontrol = false
    variable_connect_as_user = "ec2-user"
    only = [
      "amazon-ebs.amznlnx2023-rendernode-ami",
    ]
  }

  source "source.null.newuser" {
    variable_user = "deadlineuser"
    variable_uid = local.deadlineuser_uid
    add_to_group_syscontrol = false
    variable_connect_as_user = "ubuntu"
    only = [
      "amazon-ebs.deadline-db-ubuntu18-ami",
    ]
  }
}
6

nope, I spoke to soon, this just ran silently. I’m gonna dig into this a bit since at least its not failing completely.

7

I realised this should be able to be simpler, since the ssh username is defined in the amazon-ebs source blocks. according to the docs, ssh_username should be able to be referenced like I’m doing below, but this does not work if I have an ansible provisioner in a build block. Removing the source.ssh_username works fine though.

  provisioner "ansible" { # Add user deployuser
    playbook_file = "./ansible/newuser.yaml"
    user          = "${source.ssh_username}"
    extra_arguments = [
      "-v",
      "--extra-vars",
      "variable_user=deployuser sudo=true passwordless_sudo=true add_to_group_syscontrol=true variable_connect_as_user=${source.ssh_username} variable_uid=${local.deployuser_uid} syscontrol_gid=${local.syscontrol_gid} variable_host=default delegate_host=localhost"
    ]
    collections_path = "./ansible/collections"
    roles_path       = "./ansible/roles"
    ansible_env_vars = ["ANSIBLE_CONFIG=ansible/ansible.cfg"]
    galaxy_file      = "./requirements.yml"
    only = [
      "amazon-ebs.rocky8-rendernode-ami",
      "amazon-ebs.amznlnx2023-rendernode-ami",
      "amazon-ebs.deadline-db-ubuntu18-ami",
    ]
  }

we get:

Error: Failed preparing provisioner-block "ansible" ""

  on /tmp/firehawk-infra/modules/packer-firehawk-amis/modules/firehawk-ami/firehawk-ami.pkr.hcl line 841:
  (source code not available)

/tmp/firehawk-infra/modules/packer-firehawk-amis/modules/firehawk-ami/firehawk-ami.pkr.hcl:843,30-43:
Unsupported attribute; This object does not have an attribute named
"ssh_username"., and 1 other diagnostic(s)
8

And if I try to do this in the build block (and remove the var from the amazon-ebs source)

  source "amazon-ebs.rocky8-ami" {
    ssh_username = "rocky"
  }
  source "amazon-ebs.rocky8-rendernode-ami" {
    ssh_username = "rocky"
  }
  source "amazon-ebs.amznlnx2023-ami" {
    ssh_username = "ec2-user"
  }
  source "amazon-ebs.amznlnx2023-rendernode-ami" {
    ssh_username = "ec2-user"
  }
  source "amazon-ebs.ubuntu18-ami" {
    ssh_username = "ubuntu"
  }
  source "amazon-ebs.deadline-db-ubuntu18-ami" {
    ssh_username = "ubuntu"
  }

Ill get this:

Error: Failed preparing provisioner-block "ansible" ""

  on /tmp/firehawk-infra/modules/packer-firehawk-amis/modules/firehawk-ami/firehawk-ami.pkr.hcl line 859:
  (source code not available)

/tmp/firehawk-infra/modules/packer-firehawk-amis/modules/firehawk-ami/firehawk-ami.pkr.hcl:865,128-141:
Unsupported attribute; This object does not have an attribute named
"ssh_username"., and 1 other diagnostic(s)

I dont understand why both of these examples dont work.

9

I thought I’d try another example that should work according to the docs, by specifying a new var name when the source is defined at the top of the build block.

build {

  source "amazon-ebs.rocky8-ami" {
    connection_user = "rocky"
  }
  source "amazon-ebs.rocky8-rendernode-ami" {
    connection_user = "rocky"
  }
  source "amazon-ebs.amznlnx2023-ami" {
    connection_user = "ec2-user"
  }
  source "amazon-ebs.amznlnx2023-rendernode-ami" {
    connection_user = "ec2-user"
  }
  source "amazon-ebs.ubuntu18-ami" {
    connection_user = "ubuntu"
  }
  source "amazon-ebs.deadline-db-ubuntu18-ami" {
    connection_user = "ubuntu"
  }

The intention then would be for me to use that, but I get:

Installed plugin github.com/hashicorp/amazon v1.3.3 in "/root/.config/packer/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.3.3_x5.0_linux_amd64"
Installed plugin github.com/hashicorp/ansible v1.1.2 in "/root/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.2_x5.0_linux_amd64"
Error: Unsupported argument

  on /tmp/firehawk-infra/modules/packer-firehawk-amis/modules/firehawk-ami/firehawk-ami.pkr.hcl line 529:
  (source code not available)

An argument named "connection_user" is not expected here.

So is it not possible to create reusable code blocks or have variables per source block in packer? The documentation led me to beleive this should be possible.

10

I created a ticket on github for two of the above issues.