Extending Golden Images for KVM/QEMU

Packer
1

I’m looking for a solution to create golden images for KVM/QEMU/Libvirt that can be cloned, extended and re-provisioned to have different layers in a build process. I’ve managed to create a base Ubuntu Image from an ISO, which outputs a qcow file.

What I’d now like to do is use the qcow image as a source file to create new qcow files provisioned with shell and Ansible in JSON. I know that HC prefers their own format, though JSON is still part of my workflow.

I can’t seem to find anything online that helps accomplish this.
Everything I’ve tried so far has not allowed step 2 to be successful.

2

Hey there! Would disk_image and use_backing_file work for your case? With that you can provide a qcow file and packer will clone it and use it as a source instead of an ISO.

3

I am using your suggested options, no dice.
Here is the JSON file I’m using to build the stage 2 image.

{
  "builders": [
    {
      "accelerator": "kvm",
      "boot_wait": "20s",
      "disk_image": true,
      "disk_interface": "virtio",
	  "disk_size": "16385",
      "format": "qcow2",
	  "headless": true,
      "iso_checksum": "sha256:db67609278f56933db76f26962ea0ecafdfd9268910f6833839a8fd3f100bde1",
      "iso_url": "output-ubuntu1804/ubuntu1804",
      "output_directory": "output-ubuntu-layer2",
      "qemuargs": [
        [ "-m", "512" ],
		[ "-display", "none" ],
		[ "-machine", "accel=kvm" ],
        [ "-cpu", "host" ],
        [ "-smp", "cpus=1" ],
		[ "-netdev", "user,hostfwd=tcp::{{ .SSHHostPort }}-:22,id=forward"],
		[ "-device", "virtio-net,netdev=forward,id=net0"]
      ],
      "shutdown_command": "shutdown -P now",
      "ssh_password": "vagrant",
      "ssh_timeout": "1800s",
      "ssh_username": "vagrant",
      "type": "qemu",
      "use_backing_file": true,
      "vm_name": "ubuntu-layer2"
    }
  ],
  "provisioners": [
    {
      "inline": [ "sudo apt install -y nginx" ],
      "pause_before": "5s",
      "type": "shell"
    }
  ]
}

Here is the output of “PACKER_LOG=1 packer build layer2.json”.

=> qemu: Overriding default Qemu arguments with qemuargs template option...
2021/03/19 03:28:27 packer-builder-qemu plugin: Executing /usr/bin/qemu-system-x86_64: []string{"-boot", "c", "-name", "ubuntu-layer2", "-m", "512", "-display", "none", "-machine", "accel=kvm", "-smp", "cpus=1", "-netdev", "user,hostfwd=tcp::3088-:22,id=forward", "-device", "virtio-net,netdev=forward,id=net0", "-cpu", "host", "-vnc", "127.0.0.1:13", "-drive", "file=output-ubuntu-layer2/ubuntu-layer2,if=virtio,cache=writeback,discard=ignore,format=qcow2", "-drive", "file=output-ubuntu-layer2/ubuntu-layer2,if=virtio,cache=writeback,discard=ignore,format=qcow2"}
2021/03/19 03:28:27 packer-builder-qemu plugin: Started Qemu. Pid: 35502
2021/03/19 03:28:27 packer-builder-qemu plugin: Qemu stderr: qemu-system-x86_64: -drive file=output-ubuntu-layer2/ubuntu-layer2,if=virtio,cache=writeback,discard=ignore,format=qcow2: Failed to get "write" lock
2021/03/19 03:28:27 packer-builder-qemu plugin: Qemu stderr: Is another process using the image [output-ubuntu-layer2/ubuntu-layer2]?
2021/03/19 03:28:27 packer-builder-qemu plugin: failed to unlock port lockfile: close tcp 127.0.0.1:5913: use of closed network connection
2021/03/19 03:28:27 packer-builder-qemu plugin: failed to unlock port lockfile: close tcp 127.0.0.1:3088: use of closed network connection

I don’t understand why this is happening, but I assume it has something to do with networking?

Thanks for a quick response. This has been a very stressful 2 days.

4

Problem seems to be with the version of Packer in the Arch Linux User Repository. That’s a different issue in and of itself.

I compiled Packer from source on the dev branch and everything worked fine.

1 Like
5

Sad to know the two stressful days you had but good to hear o managed to get it working!

Which version were you using before trying the dev one? Could you share the issue with me so that I can investigate it?

6

Not sure if you’ve already solved this, but it can’t write properly to

output-ubuntu-layer2/ubuntu-layer2

because it’s already existing and owned by you.

7

May I ask how you solved the chaining of the builds? How do you ensure that all golden images at layer-x are rebuild if you need to change something in layer-0.

I made it somewhat work by using Jenkins as a wrapper:

  • Stage: Build layer-0:
    – Check if layer-0 exists in artifactory
    – (Re-)Build layer-0 unless it could be found
    – Publish layer-0 to artifactory unless it could be found
  • Stage: Build layer-1:
    – Download layer-0
    – Build layer-1
    – Publish layer-1
    …

But this seems not to be a very nice solution and does not really scale. I would be super happy to see something from packer itself?

8

I had to codify that logic in the scripts that were orchestrating the builds. It doesn’t necessarily rebuild all the subsequent images by force because I only have 4 total layers. The layers near the top are rebuilt at a much higher frequency, so they just pull in the changes from the base images as they are built again.