Hi guys
I am facing the below situation： I create some vms on vcenter platform using terraform, each vm has several volumes. Now I want
to update the image of vm , I use change template to achive this goal in main.tf file. But I want to remain the orginal volumes. So
what I did is like below
1 using govc remove and keep the orginal disk.
2 recreate the mvs using the terraform
3 delete the new vm’s volume and restore the orginal disk
but I found that the terraform awalys know I change/delete/move the volume. Even if I add the below parameters in ignore
lifecycle {
ignore_changes = [
“enable_disk_uuid”,
“vapp.#”,
“vapp.0.properties.%”,
“vapp.0.properties.ncm-installer.public-keys”,
“disk.#”,
“disk.0.attach”,
“disk.0.datastore_id”,
“disk.0.disk_mode”,
“disk.0.disk_sharing”,
“disk.0.eagerly_scrub”,
“disk.0.io_limit”,
“disk.0.io_reservation”,
“disk.0.io_share_count”,
“disk.0.io_share_level”,
“disk.0.keep_on_remove”,
“disk.0.io_limit”,
“disk.0.io_reservation”,
“disk.0.io_share_count”,
“disk.0.io_share_level”,
“disk.0.keep_on_remove”,
“disk.0.key”,
“disk.0.size”,
“disk.0.thin_provisioned”,
“disk.0.unit_number”,
“disk.0.write_through”,
“disk.1.attach”,
“disk.1.datastore_id”,
“disk.1.disk_mode”,
“disk.1.disk_sharing”,
“disk.1.eagerly_scrub”,
“disk.1.io_limit”,
“disk.1.io_reservation”,
“disk.1.io_share_count”,
“disk.1.io_share_level”,
“disk.1.keep_on_remove”,
“disk.1.key”,
“disk.1.size”,
“disk.1.thin_provisioned”,
“disk.1.unit_number”,
“disk.1.write_through”
]
}
That make the terraform always added the volumes back . and I will get the below error message :
2019-08-16 06:09:42,969 INFO:
Log file /opt/clcm/log/clcm_20190816_060942.log
2019-08-16 06:09:42,969 INFO:
-------------- VCENTER UPDATE IMAGE --------------
2019-08-16 06:09:42,969 INFO: Running (u’cp /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf.clcm-01 /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf’,) {‘shell’: True}
2019-08-16 06:09:44,154 INFO: Running (‘touch /root/CSF-CLCM/terraform/vcenter/resources/keystone.rc’,) {‘shell’: True}
2019-08-16 06:09:44,663 INFO: Running (‘chmod 644 /root/CSF-CLCM/terraform/vcenter/resources/keystone.rc’,) {‘shell’: True}
2019-08-16 06:09:45,177 INFO: Running (‘source /root/CSF-CLCM/terraform/vcenter/resources/keystone.rc;govc about’,) {‘shell’: True}
2019-08-16 06:09:45,260 DEBUG: Name: VMware vCenter Server
2019-08-16 06:09:45,260 DEBUG: Vendor: VMware, Inc.
2019-08-16 06:09:45,261 DEBUG: Version: 6.5.0
2019-08-16 06:09:45,261 DEBUG: Build: 10964411
2019-08-16 06:09:45,261 DEBUG: OS type: linux-x64
2019-08-16 06:09:45,262 DEBUG: API type: VirtualCenter
2019-08-16 06:09:45,262 DEBUG: API version: 6.5
2019-08-16 06:09:45,262 DEBUG: Product ID: vpx
2019-08-16 06:09:45,263 DEBUG: UUID: 9add80ad-8f50-446b-bb4c-4bed404847a2
2019-08-16 06:09:45,766 INFO: Running (‘govc about’,) {‘shell’: True}
2019-08-16 06:09:45,843 DEBUG: Name: VMware vCenter Server
2019-08-16 06:09:45,843 DEBUG: Vendor: VMware, Inc.
2019-08-16 06:09:45,844 DEBUG: Version: 6.5.0
2019-08-16 06:09:45,844 DEBUG: Build: 10964411
2019-08-16 06:09:45,844 DEBUG: OS type: linux-x64
2019-08-16 06:09:45,844 DEBUG: API type: VirtualCenter
2019-08-16 06:09:45,845 DEBUG: API version: 6.5
2019-08-16 06:09:45,845 DEBUG: Product ID: vpx
2019-08-16 06:09:45,845 DEBUG: UUID: 9add80ad-8f50-446b-bb4c-4bed404847a2
2019-08-16 06:09:47,159 INFO: Running (u"grep ‘device_name’ /root/CSF-CLCM/terraform/vcenter/resources/ip_addresses_file.clcm-01",) {‘shell’: True}
2019-08-16 06:09:47,168 DEBUG: “device_name”: “eth0”,
2019-08-16 06:09:47,168 DEBUG: “device_name”: “eth1”,
2019-08-16 06:09:47,168 DEBUG: “device_name”: “eth0”,
2019-08-16 06:09:47,168 DEBUG: “device_name”: “eth1”,
2019-08-16 06:09:47,835 INFO: Running (‘touch /root/CSF-CLCM/terraform/vcenter/resources/keystone.rc’,) {‘shell’: True}
2019-08-16 06:09:48,347 INFO: Running (‘chmod 644 /root/CSF-CLCM/terraform/vcenter/resources/keystone.rc’,) {‘shell’: True}
2019-08-16 06:09:48,861 INFO: Running (‘source /root/CSF-CLCM/terraform/vcenter/resources/keystone.rc;govc about’,) {‘shell’: True}
2019-08-16 06:09:48,941 DEBUG: Name: VMware vCenter Server
2019-08-16 06:09:48,942 DEBUG: Vendor: VMware, Inc.
2019-08-16 06:09:48,942 DEBUG: Version: 6.5.0
2019-08-16 06:09:48,942 DEBUG: Build: 10964411
2019-08-16 06:09:48,942 DEBUG: OS type: linux-x64
2019-08-16 06:09:48,942 DEBUG: API type: VirtualCenter
2019-08-16 06:09:48,943 DEBUG: API version: 6.5
2019-08-16 06:09:48,943 DEBUG: Product ID: vpx
2019-08-16 06:09:48,943 DEBUG: UUID: 9add80ad-8f50-446b-bb4c-4bed404847a2
2019-08-16 06:09:49,444 INFO: Running (‘govc about’,) {‘shell’: True}
2019-08-16 06:09:49,522 DEBUG: Name: VMware vCenter Server
2019-08-16 06:09:49,523 DEBUG: Vendor: VMware, Inc.
2019-08-16 06:09:49,523 DEBUG: Version: 6.5.0
2019-08-16 06:09:49,523 DEBUG: Build: 10964411
2019-08-16 06:09:49,523 DEBUG: OS type: linux-x64
2019-08-16 06:09:49,523 DEBUG: API type: VirtualCenter
2019-08-16 06:09:49,524 DEBUG: API version: 6.5
2019-08-16 06:09:49,524 DEBUG: Product ID: vpx
2019-08-16 06:09:49,524 DEBUG: UUID: 9add80ad-8f50-446b-bb4c-4bed404847a2
2019-08-16 06:09:51,084 INFO: Running (‘govc vm.power -dc Datacenter1 -off larryx-group1-02’,) {‘shell’: True}
2019-08-16 06:09:52,084 DEBUG: Powering off VirtualMachine:vm-274… OK
2019-08-16 06:09:52,976 INFO: Running (‘govc vm.disk.change -dc Datacenter1 -vm larryx-group1-02 -disk.name “disk-1000-0” -mode independent_persistent’,) {‘shell’: True}
2019-08-16 06:09:54,305 INFO: Running (‘govc vm.disk.change -dc Datacenter1 -vm larryx-group1-02 -disk.name “disk-1000-1” -mode independent_persistent’,) {‘shell’: True}
2019-08-16 06:09:55,969 INFO: Running (‘govc device.remove -dc Datacenter1 -vm larryx-group1-02 -keep disk-1000-0’,) {‘shell’: True}
2019-08-16 06:09:57,229 INFO: Running (‘govc device.remove -dc Datacenter1 -vm larryx-group1-02 -keep disk-1000-1’,) {‘shell’: True}
2019-08-16 06:09:58,924 INFO: Running (‘govc datastore.mkdir -dc Datacenter1 tmp’,) {‘shell’: True}
2019-08-16 06:10:01,989 INFO: Running (‘govc datastore.mv -dc Datacenter1 -f larryx-group1-02/larryx-group1-02_1.vmdk tmp/larryx-group1-02_1.vmdk’,) {‘shell’: True}
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:02] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_1.vmdk to [vsanDatastore] tmp/larryx-group1-02_1.vmdk…OK
2019-08-16 06:10:03,437 INFO: Running (‘govc datastore.mv -dc Datacenter1 -f larryx-group1-02/larryx-group1-02_2.vmdk tmp/larryx-group1-02_2.vmdk’,) {‘shell’: True}
[16-08-19 06:10:03] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:03] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:03] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:03] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:04] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:04] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:04] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/lar
[16-08-19 06:10:04] Moving [vsanDatastore] larryx-group1-02/larryx-group1-02_2.vmdk to [vsanDatastore] tmp/larryx-group1-02_2.vmdk…OK
2019-08-16 06:10:04,765 INFO: Running (‘cp /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tfbk’,) {‘shell’: True}
2019-08-16 06:10:05,280 INFO: Running (‘cp /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tfbk’,) {‘shell’: True}
2019-08-16 06:10:05,795 INFO: Running (u’sed -i “/“larryx-group1-02”/,/^\}/ d” /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf’,) {‘shell’: True}
2019-08-16 06:10:06,325 INFO: original_module count: 3
2019-08-16 06:10:06,337 INFO: current_module count: 2
2019-08-16 06:10:06,337 INFO: Running (u’diff /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf /root/CSF-CLCM/terraform/vcenter/resources/vcenter.tf.clcm-01’,) {‘shell’: True}
2019-08-16 06:10:06,348 DEBUG: 41a42,70
2019-08-16 06:10:06,348 DEBUG: > module “larryx-group1-02”
2019-08-16 06:10:06,349 DEBUG: > {
2019-08-16 06:10:06,349 DEBUG: > source = “./modules/instance/clcm-01/group_01/”
2019-08-16 06:10:06,349 DEBUG: > hostname = “larryx-group1-02”
2019-08-16 06:10:06,349 DEBUG: > count_index = 1
2019-08-16 06:10:06,350 DEBUG: > datacenter = “Datacenter1”
2019-08-16 06:10:06,350 DEBUG: > cluster = “Cluster1”
2019-08-16 06:10:06,350 DEBUG: > datastore = “vsanDatastore”
2019-08-16 06:10:06,350 DEBUG: > pool = “nancy-pool”
2019-08-16 06:10:06,351 DEBUG: > template = “bcmt-19.03.1-template”
2019-08-16 06:10:06,351 DEBUG: > network_label_net-01 = “Private-DPortGroup-1”
2019-08-16 06:10:06,351 DEBUG: > NIC_0_ipv4 = “172.16.1.121 172.16.1.122”
2019-08-16 06:10:06,351 DEBUG: > network_label_net-03 = “OAM-DPortGroup”
2019-08-16 06:10:06,351 DEBUG: > NIC_1_ipv4 = “135.1.87.107 135.1.87.108”
2019-08-16 06:10:06,352 DEBUG: >
2019-08-16 06:10:06,352 DEBUG: > ssh_user = “cloud-user”
2019-08-16 06:10:06,352 DEBUG: > ssh_key = “/opt/clcm/certs/bcmt_pri_key”
2019-08-16 06:10:06,352 DEBUG: > domain = “lucent.com”
2019-08-16 06:10:06,353 DEBUG: > dns_server1 = “135.119.144.222”
2019-08-16 06:10:06,353 DEBUG: > dns_server2 = “135.119.144.254”
2019-08-16 06:10:06,353 DEBUG: >
2019-08-16 06:10:06,353 DEBUG: > vm_offset = “1”
2019-08-16 06:10:06,353 DEBUG: > vm_role = “control storage”
2019-08-16 06:10:06,354 DEBUG: > vm_cpu = “4”
2019-08-16 06:10:06,354 DEBUG: > vm_ram = “8192”
2019-08-16 06:10:06,354 DEBUG: >
2019-08-16 06:10:06,354 DEBUG: > volume-01_size = “20”
2019-08-16 06:10:06,355 DEBUG: > volume-02_size = “10”
2019-08-16 06:10:06,355 DEBUG: > }
2019-08-16 06:10:07,169 INFO: Running (‘rm -rf /root/CSF-CLCM/terraform/vcenter/resources/.terraform’,) {‘shell’: True}
2019-08-16 06:10:07,678 INFO:
2019-08-16 06:10:07,680 INFO: Terraform heal vms step1:
2019-08-16 06:10:07,680 INFO: Running terraform init -lock=false && terraform apply -lock=false -auto-approve -no-color -state=/root/CSF-CLCM/terraform/vcenter/resources/terraform.tfstate.clcm-01
2019-08-16 06:10:07,681 INFO: Running (‘terraform init -lock=false && terraform apply -lock=false -auto-approve -no-color -state=/root/CSF-CLCM/terraform/vcenter/resources/terraform.tfstate.clcm-01’,) {‘shell’: True}
2019-08-16 06:10:07,755 DEBUG: Initializing modules…
2019-08-16 06:10:07,758 DEBUG: - module.larryx-group1-01
2019-08-16 06:10:07,759 DEBUG: Getting source “./modules/instance/clcm-01/group_01/”
2019-08-16 06:10:07,766 DEBUG: - module.larryx-group2-01
2019-08-16 06:10:07,767 DEBUG: Getting source “./modules/instance/clcm-01/group_02/”
2019-08-16 06:10:07,797 DEBUG:
2019-08-16 06:10:07,797 DEBUG: Initializing provider plugins…
2019-08-16 06:10:08,182 DEBUG:
2019-08-16 06:10:08,182 DEBUG: The following providers do not have any version constraints in configuration,
2019-08-16 06:10:08,182 DEBUG: so the latest version was installed.
2019-08-16 06:10:08,183 DEBUG:
2019-08-16 06:10:08,183 DEBUG: To prevent automatic upgrades to new major versions that may contain breaking
2019-08-16 06:10:08,183 DEBUG: changes, it is recommended to add version = “…” constraints to the
2019-08-16 06:10:08,183 DEBUG: corresponding provider blocks in configuration, with the constraint strings
2019-08-16 06:10:08,184 DEBUG: suggested below.
2019-08-16 06:10:08,184 DEBUG:
2019-08-16 06:10:08,184 DEBUG: * provider.vsphere: version = “~> 1.12”
2019-08-16 06:10:08,184 DEBUG:
2019-08-16 06:10:08,184 DEBUG: Terraform has been successfully initialized!
2019-08-16 06:10:08,185 DEBUG:
2019-08-16 06:10:08,185 DEBUG: You may now begin working with Terraform. Try running “terraform plan” to see
2019-08-16 06:10:08,185 DEBUG: any changes that are required for your infrastructure. All Terraform commands
2019-08-16 06:10:08,185 DEBUG: should now work.
2019-08-16 06:10:08,185 DEBUG:
2019-08-16 06:10:08,185 DEBUG: If you ever set or change modules or backend configuration for Terraform,
2019-08-16 06:10:08,185 DEBUG: rerun this command to reinitialize your working directory. If you forget, other
2019-08-16 06:10:08,185 DEBUG: commands will detect it and remind you to do so if necessary.
2019-08-16 06:10:09,960 DEBUG: vsphere_virtual_machine.vm-node: Refreshing state… (ID: 421fedaf-0b47-0aa4-0737-2b5de4655018)
2019-08-16 06:10:09,979 DEBUG: data.vsphere_datacenter.dc: Refreshing state…
2019-08-16 06:10:09,980 DEBUG: data.vsphere_datacenter.dc: Refreshing state…
2019-08-16 06:10:10,018 DEBUG: data.vsphere_resource_pool.pool: Refreshing state…
2019-08-16 06:10:10,032 DEBUG: data.vsphere_datastore.datastore: Refreshing state…
2019-08-16 06:10:10,044 DEBUG: data.vsphere_network.network_label_net-01: Refreshing state…
2019-08-16 06:10:10,050 DEBUG: data.vsphere_network.network_label_net-03: Refreshing state…
2019-08-16 06:10:10,057 DEBUG: data.vsphere_virtual_machine.app_template: Refreshing state…
2019-08-16 06:10:10,076 DEBUG: data.vsphere_compute_cluster.cluster: Refreshing state…
2019-08-16 06:10:10,077 DEBUG: data.vsphere_resource_pool.pool: Refreshing state…
2019-08-16 06:10:10,078 DEBUG: data.vsphere_datastore.datastore: Refreshing state…
2019-08-16 06:10:10,079 DEBUG: data.vsphere_compute_cluster.cluster: Refreshing state…
2019-08-16 06:10:10,138 DEBUG: data.vsphere_network.network_label_net-01: Refreshing state…
2019-08-16 06:10:10,159 DEBUG: data.vsphere_network.network_label_net-03: Refreshing state…
2019-08-16 06:10:10,165 DEBUG: data.vsphere_virtual_machine.app_template: Refreshing state…
2019-08-16 06:10:10,234 DEBUG: vsphere_virtual_machine.vm-node: Refreshing state… (ID: 421f4d43-6c9d-e40a-e381-9cd930024aad)
2019-08-16 06:10:10,260 DEBUG: vsphere_virtual_machine.vm-node: Refreshing state… (ID: 421f9b80-21c1-a0ff-2c38-4d4d8f6f5e9d)
2019-08-16 06:10:12,414 DEBUG: module.larryx-group1-02.vsphere_virtual_machine.vm-node: Destroying… (ID: 421fedaf-0b47-0aa4-0737-2b5de4655018)
2019-08-16 06:10:12,419 DEBUG: module.larryx-group1-01.vsphere_virtual_machine.vm-node: Destroying… (ID: 421f4d43-6c9d-e40a-e381-9cd930024aad)
2019-08-16 06:10:13,240 DEBUG: module.larryx-group1-02.vsphere_virtual_machine.vm-node: Destruction complete after 1s
2019-08-16 06:10:19,849 DEBUG: module.larryx-group1-01.vsphere_virtual_machine.vm-node: Destruction complete after 8s
2019-08-16 06:10:20,469 DEBUG:
2019-08-16 06:10:20,469 DEBUG: Error: Error applying plan:
2019-08-16 06:10:20,469 DEBUG:
2019-08-16 06:10:20,469 DEBUG: 1 error occurred:
2019-08-16 06:10:20,470 DEBUG: * module.larryx-group1-01.vsphere_virtual_machine.vm-node: vsphere_virtual_machine.vm-node: diffs didn’t match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.
2019-08-16 06:10:20,470 DEBUG:
2019-08-16 06:10:20,470 DEBUG: Please include the following information in your report:
2019-08-16 06:10:20,470 DEBUG:
2019-08-16 06:10:20,471 DEBUG: Terraform Version: 0.11.14
2019-08-16 06:10:20,471 DEBUG: Resource ID: vsphere_virtual_machine.vm-node
2019-08-16 06:10:20,471 DEBUG: Mismatch reason: attribute mismatch: disk.2.attach
2019-08-16 06:10:20,595 DEBUG:
2019-08-16 06:10:20,595 DEBUG: Also include as much context as you can about your config, state, and the steps you performed to trigger this error.
2019-08-16 06:10:20,595 DEBUG:
2019-08-16 06:10:20,595 DEBUG:
2019-08-16 06:10:20,595 DEBUG:
2019-08-16 06:10:20,595 DEBUG:
2019-08-16 06:10:20,596 DEBUG: Terraform does not automatically rollback in the face of errors.
2019-08-16 06:10:20,596 DEBUG: Instead, your Terraform state file has been partially updated with
2019-08-16 06:10:20,596 DEBUG: any resources that successfully completed. Please address the error
2019-08-16 06:10:20,596 DEBUG: above and apply again to incrementally change your infrastructure.
2019-08-16 06:10:20,596 DEBUG:
2019-08-16 06:10:20,597 DEBUG:
2019-08-16 06:10:22,735 INFO:
Terraform heal vms step1 failed. Please check the log file /opt/clcm/log/clcm_20190816_060942.log
So could you help me that how could I achieve this image update goal？
Thank in advance.