vSphere Provider - Deploy vCloud Appliance from OVA

tlb1galaxy · August 19, 2021, 7:13pm

Hello,
Relatively new to Terraform collections.
Have a question/issue around the vSphere provider → 2.0.2 and deploying the vCloud appliance.

Environment:

Terraform version: 1.0.2
vSphere provider: 2.0.2
vCenter: 7.0.2
ESXi: 7.0.2
OVA file: VMware_vCloud_Director-9.7.0.5223-17135839_OVF10.ova and/or VMware_Cloud_Director-10.2.2.5821-17855679_OVF10.ova
Terraform server: Ubuntu 21.04

Error:

module.vcda_initial_db_master.vsphere_virtual_machine.vcda01: Creating...
module.vcda_initial_db_master.vsphere_virtual_machine.vcda01: Still creating... [10s elapsed]
module.vcda_initial_db_master.vsphere_virtual_machine.vcda01: Still creating... [20s elapsed]
module.vcda_initial_db_master.vsphere_virtual_machine.vcda01: Still creating... [30s elapsed]
module.vcda_initial_db_master.vsphere_virtual_machine.vcda01: Still creating... [40s elapsed]
╷
│ Error: error while creating vapp properties config vApp property with userConfigurable=false specified in vapp.properties: [sys_name db_ha_type vmname ceip_enabled inst_id routes0 admin_email searchpath gateway ip1 netmask0 admin_fname routes1 domain db_pwd netmask1 admin_uname ip0 DNS admin_pwd]
│ 
│   with module.vcda_initial_db_master.vsphere_virtual_machine.vcda01,
│   on modules/vcda_initial_db_master/main.tf line 37, in resource "vsphere_virtual_machine" "vcda01":
│   37: resource "vsphere_virtual_machine" "vcda01" {

Questions:

Can the vsphere provider actually deploy the vCloud (VCDA) appliance OVA?
1a. From what I understand in the provider docs (vsphere_docs), it should be able to deploy the OVA with the builtin vApp properties.
How can you verify what the vApp properties syntax names are?
2a. From what I can gather, I was able to export the properties from an existing deployment (will attach below). And confirmed them via William Lam’s article to export them.

Observation/assumption:
It seems that the OVA gets deployed with basic/default settings and does not actually take any of the set vApp properties.

Code snippets:

Existing VCDA deployment vApp properties

<?xml version="1.0" encoding="UTF-8"?>
<Environment
     xmlns="http://schemas.dmtf.org/ovf/environment/1"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xmlns:oe="http://schemas.dmtf.org/ovf/environment/1"
     xmlns:ve="http://www.vmware.com/schema/ovfenv"
     oe:id=""
     ve:vCenterId="vm-6498">
   <PlatformSection>
      <Kind>VMware ESXi</Kind>
      <Version>7.0.2</Version>
      <Vendor>VMware, Inc.</Vendor>
      <Locale>en</Locale>
   </PlatformSection>
   <PropertySection>
         <Property oe:key="vami.DNS.VMware_vCloud_Director" oe:value="10.10.10.1,10.10.10.2"/>
         <Property oe:key="vami.domain.VMware_vCloud_Director" oe:value="site1.company.net"/>
         <Property oe:key="vami.gateway.VMware_vCloud_Director" oe:value="10.10.10.254"/>
         <Property oe:key="vami.ip0.VMware_vCloud_Director" oe:value="10.10.10.10"/>
         <Property oe:key="vami.ip1.VMware_vCloud_Director" oe:value="10.10.11.10"/>
         <Property oe:key="vami.netmask0.VMware_vCloud_Director" oe:value="255.255.255.0"/>
         <Property oe:key="vami.netmask1.VMware_vCloud_Director" oe:value="255.255.255.0"/>
         <Property oe:key="vami.searchpath.VMware_vCloud_Director" oe:value="site1.company.net"/>
         <Property oe:key="vcloudapp.ceip_enabled.VMware_vCloud_Director" oe:value="False"/>
         <Property oe:key="vcloudapp.db_ha_type.VMware_vCloud_Director" oe:value="primary"/>
         <Property oe:key="vcloudapp.enable_ssh.VMware_vCloud_Director" oe:value="True"/>
         <Property oe:key="vcloudapp.expire_root_password.VMware_vCloud_Director" oe:value="False"/>
         <Property oe:key="vcloudapp.nfs_mount.VMware_vCloud_Director" oe:value="10.10.10.3:/storage/lvm_transfer/exports/transfer01"/>
         <Property oe:key="vcloudapp.ntp-server.VMware_vCloud_Director" oe:value="10.10.10.254"/>
         <Property oe:key="vcloudapp.varoot-password.VMware_vCloud_Director" oe:value=""/>
         <Property oe:key="vcloudconf.admin_email.VMware_vCloud_Director" oe:value="admin@company.net"/>
         <Property oe:key="vcloudconf.admin_fname.VMware_vCloud_Director" oe:value="vCD Admin"/>
         <Property oe:key="vcloudconf.admin_pwd.VMware_vCloud_Director" oe:value=""/>
         <Property oe:key="vcloudconf.admin_uname.VMware_vCloud_Director" oe:value="administrator"/>
         <Property oe:key="vcloudconf.db_pwd.VMware_vCloud_Director" oe:value="XXXXX"/>
         <Property oe:key="vcloudconf.inst_id.VMware_vCloud_Director" oe:value="1"/>
         <Property oe:key="vcloudconf.sys_name.VMware_vCloud_Director" oe:value="vcd1"/>
         <Property oe:key="vcloudnet.routes0.VMware_vCloud_Director" oe:value=""/>
         <Property oe:key="vcloudnet.routes1.VMware_vCloud_Director" oe:value=""/>
         <Property oe:key="vm.vmname" oe:value="VMware_vCloud_Director"/>
   </PropertySection>
   <ve:EthernetAdapterSection>
      <ve:Adapter ve:mac="00:50:56:bb:bb:bb" ve:network="net1" ve:unitNumber="7"/>
      <ve:Adapter ve:mac="00:50:56:bb:cc:cc" ve:network="net2" ve:unitNumber="8"/>
   </ve:EthernetAdapterSection>
</Environment>

main.tf

data "vsphere_datacenter" "datacenter" {
  name = var.vmware_vcsa_vdc_tonroot
}

data "vsphere_compute_cluster" "compute_cluster" {
  name          = var.vmware_compute_cluster_mgmtcl01
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_host" "mgmtcl01_esxi01" {
  name          = var.vmware_esxi_mgmtcl01_esxi01
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_datastore" "datastore_initial_deployment" {
  name          = var.vmware_datastore_initial_deployment
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_distributed_virtual_switch" "mgmtcl01_dvs_trunk" {
  name          = var.vmware_dvs_mgmtcl01_dvs_trunk
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_network" "network_mgmt01_net1" {
  name                            = var.vmware_dvs_portgrp_mgmt01_net1
  datacenter_id                   = data.vsphere_datacenter.datacenter.id
  distributed_virtual_switch_uuid = data.vsphere_distributed_virtual_switch.mgmtcl01_dvs_trunk.id
}

data "vsphere_network" "network_mgmt01_net2" {
  name                            = var.vmware_dvs_portgrp_mgmt01_net2
  datacenter_id                   = data.vsphere_datacenter.datacenter.id
  distributed_virtual_switch_uuid = data.vsphere_distributed_virtual_switch.mgmtcl01_dvs_trunk.id
}

resource "vsphere_virtual_machine" "vcda01" {
  name             = var.vmware_vcda_01_vmname
  resource_pool_id = data.vsphere_compute_cluster.compute_cluster.resource_pool_id
  datastore_id     = data.vsphere_datastore.datastore_initial_deployment.id
  datacenter_id    = data.vsphere_datacenter.datacenter.id
  host_system_id   = data.vsphere_host.mgmtcl01_esxi01.id

  wait_for_guest_net_timeout = 0
  wait_for_guest_ip_timeout = 0

  ovf_deploy {
    #remote_ovf_url = "https://download3.vmware.com/software/vmw-tools/nested-esxi/Nested_ESXi7.0_Appliance_Template_v1.ova"
    allow_unverified_ssl_cert = true
#    local_ovf_path    = "/storage/downloads/vmware/VMware_Cloud_Director-10.2.2.5821-17855679_OVF10.ova"
    local_ovf_path    = "/storage/downloads/vmware/VMware_vCloud_Director-9.7.0.5223-17135839_OVF10.ova"
    disk_provisioning = "thin"
#    ovf_network_map   = {
#                      "net1" = data.vsphere_network.network_mgmt01_toninfra.id
#                      "net2" = data.vsphere_network.network_mgmt01_toninfra-db.id
#    }
  }

  vapp {
    properties = {
      "ntp-server" = "10.10.10.254",
      "varoot-password" = "XXXXX",
      "expire_root_password" = "False",
      "enable_ssh" = "True",
      "routes0" = "",
      "routes1" = "",
      "gateway" = "10.10.10.254",
      "domain" = "company.net",
      "searchpath" = "company.net",
      "DNS" = "10.10.10.1,10.10.10.2",
      "ip0" = "10.10.10.10",
      "netmask0" = "255.255.255.0",
      "ip1" = "10.10.11.10",
      "netmask1" = "255.255.255.0",
      "ceip_enabled" = "False",
      "db_ha_type" = "primary",
      "nfs_mount" = "10.10.10.3:/storage/lvm_transfer/exports/transfer01",
      "admin_email" = "admin@company.net",
      "admin_fname" = "vCD Admin",
      "admin_pwd" = "XXXXX",
      "admin_uname" = "administrator",
      "db_pwd" = "XXXXX",
      "inst_id" = "2",
      "sys_name" = "vcd2",
      "vmname" = "VMware_vCloud_Director"
    }
  }
}

tlb1galaxy · August 20, 2021, 1:32pm

So figured out the underlying issue. The error message says the problem but does not specifically tell you the exact vApp option that is the problem.
So the actual vApp properties that were the problem are:
db_ha_type
vmname

These properties are set to “userConfigurable=false” and cannot be configured during deployment. Once I removed these from the vApp properties block, I was able to get the VCDA to deploy. Below is a snippet of the final code block.

Background:
To find out which OVA options are configurable or not, you need to:

expand the OVA
Open the OVF file
(if you really want to make these configurable, then you can modify the OVF and re-pack the OVA). You can follow this article on how to do this.

Code:
main.tf

data "vsphere_datacenter" "datacenter" {
  name = var.vmware_vcsa_vdc_tonroot
}

data "vsphere_compute_cluster" "compute_cluster" {
  name          = var.vmware_compute_cluster_mgmtcl01
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_host" "mgmtcl01_esxi01" {
  name          = var.vmware_esxi_mgmtcl01_esxi01
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_datastore" "datastore_initial_deployment" {
  name          = var.vmware_datastore_initial_deployment
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_distributed_virtual_switch" "mgmtcl01_dvs_trunk" {
  name          = var.vmware_dvs_mgmtcl01_dvs_trunk
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_network" "network_mgmt01_net1" {
  name                            = var.vmware_dvs_portgrp_mgmt01_net1
  datacenter_id                   = data.vsphere_datacenter.datacenter.id
  distributed_virtual_switch_uuid = data.vsphere_distributed_virtual_switch.mgmtcl01_dvs_trunk.id
}

data "vsphere_network" "network_mgmt01_net2" {
  name                            = var.vmware_dvs_portgrp_mgmt01_net2
  datacenter_id                   = data.vsphere_datacenter.datacenter.id
  distributed_virtual_switch_uuid = data.vsphere_distributed_virtual_switch.mgmtcl01_dvs_trunk.id
}

resource "vsphere_virtual_machine" "vcda01" {
  name             = var.vmware_vcda_01_vmname
  num_cpus = 4  # need to manually set as the deployment type will not
  memory = 4  # need to manually set as the deplyment type will not
  
  # 1st configured interface
  network_interface {
    network_id = data.vsphere_network.network_mgmt01_net1.id
    adapter_type = "vmxnet3"
    ovf_mapping = "eth0 Network"
  }

  # 2nd configured interface
  network_interface {
    network_id = data.vsphere_network.network_mgmt01_net2.id
    adapter_type = "vmxnet3"
    ovf_mapping = "eth1 Network"
  }

  resource_pool_id = data.vsphere_compute_cluster.compute_cluster.resource_pool_id
  datastore_id     = data.vsphere_datastore.datastore_initial_deployment.id
  datacenter_id    = data.vsphere_datacenter.datacenter.id
  host_system_id   = data.vsphere_host.mgmtcl01_esxi01.id

  wait_for_guest_net_timeout = 0
  wait_for_guest_ip_timeout = 0

  ovf_deploy {
    allow_unverified_ssl_cert = true
    local_ovf_path    = "/storage/downloads/vmware/VMware_vCloud_Director-9.7.0.5223-17135839_OVF10.ova"
    disk_provisioning = "thin"

    # Discover options via expanded OVF or manually deploying an appliance [options= 'primary-small', 'primary-large', 'standby-small', 'standby-large', 'cell']
    deployment_option = "primary-large"  


  vapp {
    properties = {
      "ntp-server" = "10.10.10.254",
      "varoot-password" = "XXXXX",
      "expire_root_password" = "False",
      "enable_ssh" = "True",
      "routes0" = "",
      "routes1" = "",
      "gateway" = "10.10.10.254",
      "domain" = "company.net",
      "searchpath" = "company.net",
      "DNS" = "10.10.10.1,10.10.10.2",
      "ip0" = "10.10.10.10",
      "netmask0" = "255.255.255.0",
      "ip1" = "10.10.11.10",
      "netmask1" = "255.255.255.0",
      "ceip_enabled" = "False",
      "nfs_mount" = "10.10.10.3:/storage/lvm_transfer/exports/transfer01",
      "admin_email" = "admin@company.net",
      "admin_fname" = "vCD Admin",
      "admin_pwd" = "XXXXX",
      "admin_uname" = "administrator",
      "db_pwd" = "XXXXX",
      "inst_id" = "2",
      "sys_name" = "vcd2"
    }
  }
}

tenthirtyam · March 18, 2022, 3:16am

Example config: terrafom-examples-vmware/vsphere/vsphere-virtual-machine/clone-ovf-vcda at main · tenthirtyam/terrafom-examples-vmware · GitHub