Vsphere-iso build fails with 404 uploading ISO

rafspeik · January 21, 2023, 8:43pm

I’ve got a very simple Packer build which fails with 404 when it tries to upload the ISO.
ESXi on the host and vCenter are both version 8. I’ve also testing with a host running ESXi 7.

The log output is pretty useless to my eyes, providing no indication of what isn’t found.

I’ve double and triple checked the name of the host, datacentre, folder, vCenter server, and datastore, along with trying alternatives for most of those, so I’m struggling to understand what could be causing the issue here.

Any ideas or insight would be much appreciated.

packer build output:

$ PACKER_LOG=1 packer build .

2023/01/21 20:25:28 [INFO] Packer version: 1.8.5 [go1.18.9 darwin arm64]
2023/01/21 20:25:28 [TRACE] discovering plugins in /opt/homebrew/bin
2023/01/21 20:25:28 [TRACE] discovering plugins in /Users/user/.config/packer/plugins
2023/01/21 20:25:28 [DEBUG] Discovered plugin: vsphere = /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64
2023/01/21 20:25:28 [INFO] found external [clone iso supervisor] builders from vsphere plugin
2023/01/21 20:25:28 [INFO] found external [-packer-default-plugin-name- template] post-processors from vsphere plugin
2023/01/21 20:25:28 [TRACE] discovering plugins in .
2023/01/21 20:25:28 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2023/01/21 20:25:28 [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/user/.packerconfig
2023/01/21 20:25:28 [WARN] Config file doesn't exist: /Users/user/.packerconfig
2023/01/21 20:25:28 [INFO] Setting cache directory: /Users/user/.cache/packer
2023/01/21 20:25:28 [TRACE] listing potential installations for "github.com/hashicorp/vsphere" that match ">= v1.1.1". plugingetter.ListInstallationsOptions{FromFolders:[]string{"/opt/homebrew/bin/packer", ".", "/Users/user/.config/packer/plugins"}, BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"darwin", ARCH:"arm64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0x140005c8300)}}}}
2023/01/21 20:25:28 [TRACE] Found the following "github.com/hashicorp/vsphere" installations: [{/Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 v1.1.1}]
2023/01/21 20:25:28 [INFO] found external [clone iso supervisor] builders from vsphere plugin
2023/01/21 20:25:28 [INFO] found external [-packer-default-plugin-name- template] post-processors from vsphere plugin
2023/01/21 20:25:28 [TRACE] Starting external plugin /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 start builder iso
2023/01/21 20:25:28 Starting plugin: /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 []string{"/Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64", "start", "builder", "iso"}
2023/01/21 20:25:28 Waiting for RPC address for: /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64
2023/01/21 20:25:28 Received unix RPC address for /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64: addr is /var/folders/vp/fy9xlglx2hl_b422g0_l4n3m0000gn/T/packer-plugin762866830
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 Plugin address: unix /var/folders/vp/fy9xlglx2hl_b422g0_l4n3m0000gn/T/packer-plugin762866830
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 Waiting for connection...
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 Serving a plugin connection...
2023/01/21 20:25:28 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:28 [TRACE] starting builder iso
2023/01/21 20:25:28 Build debug mode: false
2023/01/21 20:25:28 Force build: false
2023/01/21 20:25:28 On error:
template.vsphere-iso.debian-11: output will be in this color.

2023/01/21 20:25:28 Waiting on builds to complete...
2023/01/21 20:25:28 Starting build run: template.vsphere-iso.debian-11
2023/01/21 20:25:28 Running builder:
2023/01/21 20:25:28 [INFO] (telemetry) Starting builder vsphere-iso.debian-11
==> template.vsphere-iso.debian-11: Retrieving ISO
==> template.vsphere-iso.debian-11: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso
==> template.vsphere-iso.debian-11: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso?checksum=sha256%3Ae482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d
2023/01/21 20:25:29 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:29 Acquiring lock for: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso?checksum=sha256%3Ae482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d (/Users/user/.cache/packer/588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso.lock)
2023/01/21 20:25:29 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:29 Leaving retrieve loop for ISO
2023/01/21 20:25:29 packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64 plugin: 2023/01/21 20:25:29 No CD files specified. CD disk will not be made.
==> template.vsphere-iso.debian-11: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso?checksum=sha256%3Ae482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d => /Users/user/.cache/packer/588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso
==> template.vsphere-iso.debian-11: Uploading 588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso to packer_cache/588a8fbcdb7814b92d00d21914ddb4c8ad48a406.iso
Build 'template.vsphere-iso.debian-11' errored after 1 second 177 milliseconds: 404 Not Found

==> Wait completed after 1 second 177 milliseconds

==> Some builds didn't complete successfully and had errors:
2023/01/21 20:25:30 [INFO] (telemetry) ending vsphere-iso.debian-11
--> template.vsphere-iso.debian-11: 404 Not Found

==> Wait completed after 1 second 177 milliseconds
2023/01/21 20:25:30 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2023/01/21 20:25:30 machine readable: template.vsphere-iso.debian-11,error []string{"404 Not Found"}
==> Builds finished but no artifacts were created.
==> Builds finished but no artifacts were created.
2023/01/21 20:25:30 [INFO] (telemetry) Finalizing.
2023/01/21 20:25:30 waiting for all plugin processes to complete...
2023/01/21 20:25:30 /Users/user/.config/packer/plugins/github.com/hashicorp/vsphere/packer-plugin-vsphere_v1.1.1_x5.0_darwin_arm64: plugin process exited

Packer file (note I’m using older guest_os_type and vm_version because this template is also used on older hosts, but changing these values has no effect on the result):

packer {
  required_version = ">= 1.8.4"
  required_plugins {
    vsphere = {
      version = ">= v1.1.1"
      source  = "github.com/hashicorp/vsphere"
    }
  }
}

source "vsphere-iso" "debian-11" {
  vm_name					      = "Deb11-Test"
  guest_os_type			    = "debian10_64Guest" 
  vm_version				    = "15"
  CPUs					        = "1"
  RAM						        = "1024"
  storage {
    disk_size				      = "16384"
    disk_thin_provisioned	= true
  }
  vcenter_server		    = "vc.domain.com"
  username			        = "administrator@vc.domain.com"
  password			        = "obfuscated"
  insecure_connection	  = true
  datacenter			      = "obfuscated"
  datastore			        = "obfuscated"
  folder				        = "obfuscated"
  host				          = "obfuscated"
  iso_checksum		      = "sha256:e482910626b30f9a7de9b0cc142c3d4a079fbfa96110083be1d0b473671ce08d"
  iso_url				        = "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso"
  ssh_username		      = "obfuscated"
  ssh_password		      = "obfuscated"
}

build {
  name 	= "template"
  sources = ["source.vsphere-iso.debian-11"]
}

sestegra · January 22, 2023, 4:18pm

The 404 error comes from the vSphere API when uploading the ISO file to the datastore.
Please check vSphere logs to see more details.

rafspeik · January 22, 2023, 4:31pm

Thanks, I suspected this was the case but have been struggling to pin down exactly what’s causing the error.

I tailed the vpxd log whilst reproducing the error and this seems to be the most relevant part of the result:
Edit: Removed the log snippet, I misidentified this as being related to the issue I’m having here and later discovered that the error is logged intermittently regardless of Packer activity. I’m continuing to try to pin down an exact error for this on the vSphere side.

I’m unsure whether this is a vSphere problem or a Packer problem (possibly an environmental issue in vSphere, given I can find no reports of anyone else with this issue?), but either way, I’m not sure exactly how this is happening; I’m not referring to any clusters, as you can see in the Packer config above, and all the object names that I am referencing (i.e. datacentre, datastore, etc.) are absolutely valid.

sestegra · January 22, 2023, 6:35pm

Try to upload the ISO file in your datastore manually using govc.

See command usage

github.com

vmware/govmomi/blob/main/govc/USAGE.md#datastoreupload

# govc usage

This document is generated from `govc -h` and `govc $cmd -h` commands.

The following common options are filtered out in this document,
but appear via `govc $cmd -h`:

```
  -cert=                    Certificate [GOVC_CERTIFICATE]
  -debug=false              Store debug logs [GOVC_DEBUG]
  -trace=false              Write SOAP/REST traffic to stderr
  -verbose=false            Write request/response data to stderr
  -dump=false               Enable output dump
  -json=false               Enable JSON output
  -xml=false                Enable XML output
  -k=false                  Skip verification of server certificate [GOVC_INSECURE]
  -key=                     Private key [GOVC_PRIVATE_KEY]
  -persist-session=true     Persist session to disk [GOVC_PERSIST_SESSION]
  -tls-ca-certs=            TLS CA certificates file [GOVC_TLS_CA_CERTS]
  -tls-known-hosts=         TLS known hosts file [GOVC_TLS_KNOWN_HOSTS]

This file has been truncated. show original

rafspeik · January 22, 2023, 8:43pm

Good idea, I tried uploading through the vCenter web UI and had no problems, but this is a better test. Thanks.

Interestingly, I had the same result:

GOVC_DEBUG=true GOVC_URL=obfuscated GOVC_USERNAME=obfuscated GOVC_PASSWORD='obfuscated' GOVC_INSECURE=true govc datastore.upload -verbose=true -dc Dev -ds ESX4-Datastore01 ./testfile.zip testfile.zip
RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...group-d1 name: Datacenters

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...datacenter-26120 name: obfuscated
...datacenter-2     name: obfuscated
...datacenter-19740 name: Dev

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...datacenter-19740 datastoreFolder: Folder:group-s19743
...datacenter-19740 hostFolder:      Folder:group-h19742
...datacenter-19740 name:            Dev
...datacenter-19740 networkFolder:   Folder:group-n19744
...datacenter-19740 vmFolder:        Folder:group-v19741

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...group-s19743     name:   datastore
...group-s19743     parent: Datacenter:datacenter-19740
...datacenter-19740 name:   Dev
...datacenter-19740 parent: Folder:group-d1
...group-d1         name:   Datacenters

RetrieveProperties(propertyCollector, []types.PropertyFilterSpec)...
...datastore-33185 name: ESX4-Datastore01

[22-01-23 20:37:23] Uploading... Error: 404 Not Found
govc: 404 Not Found

Am I right in thinking that the lack of detail on what is not being found is the fault of the vSphere API here? Very frustrating.

sestegra · January 26, 2023, 1:22pm

The issue is most probably on the vSphere side around your datastore.
Check the permissions that you have with the account used by the Packer’s build?
Do you have a proxy between your machine and the vSphere server?

I tested with vSphere 7.0.3 and Packer 1.8.5 with success.

rafspeik · January 27, 2023, 10:53am

I’ve come to this conclusion too… No proxy, permissions are all valid, I’ve been testing with the default administrator account.

Seems like there’s a fairly deep-rooted issue with my vCenter appliance. This will be a fun one to deal with… /s

Anyway, thanks for your help!

IM_NZ · June 29, 2023, 2:14am

Hi, did you ever figure this one out? I think I am having the same issue.

rafspeik · June 29, 2023, 8:56am

In the end I just used the set_host_for_datastore_uploads option to upload directly to the host.

Note this hasn’t fixed the problem, I still can’t upload to a datastore through vCenter when using govc, powercli, etc., but it’s a suitable workaround for Packer itself.

IM_NZ · June 29, 2023, 11:07pm

Thanks for that. It did actually work for the upload, but unfortunately the vm won’t start because my environment is a cluster with drs enabled, so you can’t specify the host.

looks like that option is out for me.
Thanks again

Topic		Replies	Views
Vsphere-iso buid fails with 404 Not Found Packer	0	569	July 11, 2022
vSphere iso builder with datastore Packer vsphere	3	2473	August 15, 2021
Error in Packer vSphere ISO for Ubuntu 20.04.3 Packer vsphere	0	1584	September 24, 2021
Vsphere-iso upload to packer_cache hangs Packer vsphere	1	290	September 10, 2024
Packer 1.7.3 supports vsphere 7.0.2? Packer	1	536	September 7, 2021

Vsphere-iso build fails with 404 uploading ISO

Related topics