How to log into VM during build, without pausing at every step?

Packer
1

We use packer to build AWS images, and have a provisioner that runs a bunch of scripts from the local repo on the EC2 instance, like this:

    {
      "type": "shell",
      "execute_command": "sudo bash -euxo pipefail -c '{{ .Vars }} {{ .Path }}'",
      "scripts": [
        "scripts/...",
        "scripts/...",
...

For working on these scripts, or troubleshooting failures, it can be very useful to log into the temporary EC2 instance during a Packer build and inspect things, and perhaps edit the scripts and re-run. Packer has two flags that make this possible:

  • -on-error=ask : Whenever a provisioner fails - which in this case can mean one of the scripts exited with an error - the build pauses and prompts you to continue, abort, or retry. I can edit the script in my local checkout and tell packer to retry, and the subsequent run will use my newly-edited script.

  • -debug : This does two things. a) It emits a .pem file at the start of the build, containing an ssh key I can use to ssh to ec2-user@[instance]. b) It tells packer to pause after every step, asking me if I want to abort, continue, or retry.

Although this works, it’s tedious and time consuming to have to hit enter a zillion times for a bunch of steps that happen before the provisioners I care about. I want it to just run all the way until some script exits with an error, and only then pause - that’s how it works if I use -on-error=ask without -debug. But without -debug, I don’t get the ssh key, so I can’t log into the instance and inspect things there!

Is there any way to get packer to run the build without waiting for confirmation, until it hits an error, but still allow me to log into the temporary instance during the packer build?

2

Since I have not yet been able to find anyone with an answer to this question, I guess packer just can’t do this. It’s really unfortunate that the ability to emit the SSH key has been combined with all other debug functionality and can’t be enabled separately.

I thought, okay, this must be a simple change, so I went to the github repo and started trying to add the flag… but it turns out that flags are plumbed all the way through to countless functions in a lot of different versions of different builders, and I got lost in the maze. The fact that I’m not a Go developer and don’t have an IDE set up certainly doesn’t help, because it’s impossible to follow chains of single-character names using simple search or grep.

So, here’s as far as I got before I realized this would take many hours trying to understand how the code is structured:

Would anyone who actually knows the packer code be willing to take my branch and turn it into something that works?

1 Like
3

Oh god yes, please…

4

Hello gitcos,

Thanks for sharing your experience with navigating the Packer code. We understand given all of the plugin code how things can be a bit hard to follow. We hope with the new plugin split work that things will get easier.

That aside, what you are trying to achieve should be doable using a few core provisioners and build variables.

Have you used build variables or the breakpoint provisioner before?

I will include a few links on the bottom of this post. But the general idea is that you can use a shell-local provisioner to output the generated PEM file via Packer build variables, which you can then follow with a breakpoint provisioner to pause the build. The breakpoint provisioner should be added before the provisioner that is erroring.

I’ve included a HCL example but you can use the same approach for JSON builds, which seems to be what you are using given your example above.

source "amazon-ebs" "basic-example" {
  region =  "us-east-1"
  source_ami_filter {
    filters = {
       virtualization-type = "hvm"
       name = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
       root-device-type = "ebs"
    }
    owners = ["099720109477"]
    most_recent = true
  }
  instance_type =  "t2.small"
  ssh_username =  "ubuntu"
  ssh_agent_auth = false
  ami_name =  "packer_AWS_{{timestamp}}"
}

build {
  sources = [
    "source.amazon-ebs.basic-example"
  ]

  provisioner "shell-local" {
    inline = ["echo '${build.SSHPrivateKey}'", "echo '${build.SSHPrivateKey}' > /tmp/packer-aws.pem"]
  }
  // Packer will break here and wait for a user to hit <enter> to continue
  provisioner "breakpoint" {}

  // Any provisioner you wish to debug manually
}

Running packer build with the above template will create an Amazon instance, output the generated SSH Private key to a local filesystem, and pause the build until the user is ready to continue.

With the local SSH private key file you will be able to log into the remote instance via

chmod 400 /tmp/packer-aws.pem
ssh -i /tmp/packer-aws.pem ubuntu@<elastic-ip>

Please give the above configuration a try and let me know if this solution is what you are looking for.

Here are the direct links to build variables and the breakpoint provisioner.

Build variables in HCL
Build variables in JSON
Breakpoint Provisioner

5

Belated thanks for a solution. I think it would work. By July 2021 I was no longer regularly working with Packer, as that project had been handed off to a different team, so I was not able to benefit from it, but perhaps others will.

However, shouldn’t this just be a core feature of packer, with a simple flag? It’s really extremely weird that a single flag controls whether you get an ssh key, and that it stops at every step. Getting the ssh key should be possible with a separate flag that has no other effects. Or, it could just be default behavior.