Vsphere-iso build fails with 404 uploading ISO

I’ve got a very simple Packer build which fails with 404 when it tries to upload the ISO.
ESXi on the host and vCenter are both version 8. I’ve also testing with a host running ESXi 7.

The log output is pretty useless to my eyes, providing no indication of what isn’t found.

I’ve double and triple checked the name of the host, datacentre, folder, vCenter server, and datastore, along with trying alternatives for most of those, so I’m struggling to understand what could be causing the issue here.

Any ideas or insight would be much appreciated.

packer build output:

$ PACKER_LOG=1 packer build .

2023/01/21 20:25:28 [INFO] Packer version: 1.8.5 [go1.18.9 darwin arm64]
2023/01/21 20:25:28 [TRACE] discovering plugins in /opt/homebrew/bin
2023/01/21 20:25:28 [TRACE] discovering plugins in /Users/user/.config/packer/plugins
2023/01/21 20:25:28 [DEBUG] Discovered plugin: vsphere = /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64
2023/01/21 20:25:28 [INFO] found external [clone iso supervisor] builders from vsphere plugin
2023/01/21 20:25:28 [INFO] found external [-packer-default-plugin-name- template] post-processors from vsphere plugin
2023/01/21 20:25:28 [TRACE] discovering plugins in .
2023/01/21 20:25:28 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2023/01/21 20:25:28 [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/user/.packerconfig
2023/01/21 20:25:28 [WARN] Config file doesn't exist: /Users/user/.packerconfig
2023/01/21 20:25:28 [INFO] Setting cache directory: /Users/user/.cache/packer
2023/01/21 20:25:28 [TRACE] listing potential installations for "github.com/hashicorp/vsphere" that match ">= v1.1.1". plugingetter.ListInstallationsOptions{FromFolders:[]string{"/opt/homebrew/bin/packer", ".", "/Users/user/.config/packer/plugins"}, BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"darwin", ARCH:"arm64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0x140005c8300)}}}}
2023/01/21 20:25:28 [TRACE] Found the following "github.com/hashicorp/vsphere" installations: [{/Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 v1.1.1}]
2023/01/21 20:25:28 [INFO] found external [clone iso supervisor] builders from vsphere plugin
2023/01/21 20:25:28 [INFO] found external [-packer-default-plugin-name- template] post-processors from vsphere plugin
2023/01/21 20:25:28 [TRACE] Starting external plugin /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 start builder iso
2023/01/21 20:25:28 Starting plugin: /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 []string{"/Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64", "start", "builder", "iso"}
2023/01/21 20:25:28 Waiting for RPC address for: /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64
2023/01/21 20:25:28 Received unix RPC address for /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64: addr is /var/folders/vp/fy9xlglx2hl_b422g0_l4n3m0000gn/T/packer-plugin762866830
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 Plugin address: unix /var/folders/vp/fy9xlglx2hl_b422g0_l4n3m0000gn/T/packer-plugin762866830
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 Waiting for connection...
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 Serving a plugin connection...
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 [TRACE] starting builder iso
2023/01/21 20:25:28 Build debug mode: false
2023/01/21 20:25:28 Force build: false
2023/01/21 20:25:28 On error:
template.vsphere-iso.debian-11: output will be in this color.

2023/01/21 20:25:28 Waiting on builds to complete...
2023/01/21 20:25:28 Starting build run: template.vsphere-iso.debian-11
2023/01/21 20:25:28 Running builder:
2023/01/21 20:25:28 [INFO] (telemetry) Starting builder vsphere-iso.debian-11
==> template.vsphere-iso.debian-11: Retrieving ISO
==> template.vsphere-iso.debian-11: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso
==> template.vsphere-iso.debian-11: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso?checksum=sha256%3Ae482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d
2023/01/21 20:25:29 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:29 Acquiring lock for: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso?checksum=sha256%3Ae482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d (/Users/user/.cache/packer/588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso.lock)
2023/01/21 20:25:29 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:29 Leaving retrieve loop for ISO
2023/01/21 20:25:29 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:29 No CD files specified. CD disk will not be made.
==> template.vsphere-iso.debian-11: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso?checksum=sha256%3Ae482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d => /Users/user/.cache/packer/588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso
==> template.vsphere-iso.debian-11: Uploading 588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso to packer_cache/588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso
Build 'template.vsphere-iso.debian-11' errored after 1 second 177 milliseconds: 404 Not Found

==> Wait completed after 1 second 177 milliseconds

==> Some builds didn't complete successfully and had errors:
2023/01/21 20:25:30 [INFO] (telemetry) ending vsphere-iso.debian-11
--> template.vsphere-iso.debian-11: 404 Not Found

==> Wait completed after 1 second 177 milliseconds
2023/01/21 20:25:30 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2023/01/21 20:25:30 machine readable: template.vsphere-iso.debian-11,error []string{"404 Not Found"}
==> Builds finished but no artifacts were created.
==> Builds finished but no artifacts were created.
2023/01/21 20:25:30 [INFO] (telemetry) Finalizing.
2023/01/21 20:25:30 waiting for all plugin processes to complete...
2023/01/21 20:25:30 /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64: plugin process exited

Packer file (note I’m using older guest_os_type and vm_version because this template is also used on older hosts, but changing these values has no effect on the result):

packer {
  required_version = ">= 1.8.4"
  required_plugins {
    vsphere = {
      version = ">= v1.1.1"
      source  = "github.com/hashicorp/vsphere"
    }
  }
}

source "vsphere-iso" "debian-11" {
  vm_name					      = "Deb11-Test"
  guest_os_type			    = "debian10_64Guest" 
  vm_version				    = "15"
  CPUs					        = "1"
  RAM						        = "1024"
  storage {
    disk_size				      = "16384"
    disk_thin_provisioned	= true
  }
  vcenter_server		    = "vc.domain.com"
  username			        = "administrator@vc.domain.com"
  password			        = "obfuscated"
  insecure_connection	  = true
  datacenter			      = "obfuscated"
  datastore			        = "obfuscated"
  folder				        = "obfuscated"
  host				          = "obfuscated"
  iso_checksum		      = "sha256:e482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d"
  iso_url				        = "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso"
  ssh_username		      = "obfuscated"
  ssh_password		      = "obfuscated"
}

build {
  name 	= "template"
  sources = ["source.vsphere-iso.debian-11"]
}

The 404 error comes from the vSphere API when uploading the ISO file to the datastore.
Please check vSphere logs to see more details.

Thanks, I suspected this was the case but have been struggling to pin down exactly what’s causing the error.

I tailed the vpxd log whilst reproducing the error and this seems to be the most relevant part of the result:
Edit: Removed the log snippet, I misidentified this as being related to the issue I’m having here and later discovered that the error is logged intermittently regardless of Packer activity. I’m continuing to try to pin down an exact error for this on the vSphere side.

I’m unsure whether this is a vSphere problem or a Packer problem (possibly an environmental issue in vSphere, given I can find no reports of anyone else with this issue?), but either way, I’m not sure exactly how this is happening; I’m not referring to any clusters, as you can see in the Packer config above, and all the object names that I am referencing (i.e. datacentre, datastore, etc.) are absolutely valid.

Try to upload the ISO file in your datastore manually using govc.

See command usage

Good idea, I tried uploading through the vCenter web UI and had no problems, but this is a better test. Thanks.

Interestingly, I had the same result:

GOVC_DEBUG=true GOVC_URL=obfuscated GOVC_USERNAME=obfuscated GOVC_PASSWORD='obfuscated' GOVC_INSECURE=true govc datastore.upload -verbose=true -dc Dev -ds ESX4-Datastore01 ./testfile.zip testfile.zip
RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...group-d1 name: Datacenters

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...datacenter-26120 name: obfuscated
...datacenter-2     name: obfuscated
...datacenter-19740 name: Dev

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...datacenter-19740 datastoreFolder: Folder:group-s19743
...datacenter-19740 hostFolder:      Folder:group-h19742
...datacenter-19740 name:            Dev
...datacenter-19740 networkFolder:   Folder:group-n19744
...datacenter-19740 vmFolder:        Folder:group-v19741

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...group-s19743     name:   datastore
...group-s19743     parent: Datacenter:datacenter-19740
...datacenter-19740 name:   Dev
...datacenter-19740 parent: Folder:group-d1
...group-d1         name:   Datacenters

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...datastore-33185 name: ESX4-Datastore01

[22-01-23 20:37:23] Uploading... Error: 404 Not Found
govc: 404 Not Found

Am I right in thinking that the lack of detail on what is not being found is the fault of the vSphere API here? Very frustrating.

The issue is most probably on the vSphere side around your datastore.
Check the permissions that you have with the account used by the Packer’s build?
Do you have a proxy between your machine and the vSphere server?

I tested with vSphere 7.0.3 and Packer 1.8.5 with success.

I’ve come to this conclusion too… No proxy, permissions are all valid, I’ve been testing with the default administrator account.

Seems like there’s a fairly deep-rooted issue with my vCenter appliance. This will be a fun one to deal with… /s

Anyway, thanks for your help!