What is the correct way to deploy and upgrade virtual machines in the Azure Cloud using Terraform Azure Provider?
- I use Packer to bake an image with all required linux packages. This step is executed in the first job of Release pipeline in Azure DevOps (with Azure-provided agents).
- On the next task I deploy the entire infrastructure via Terraform where the key point is usage of
azurerm_linux_virtual_machine_scale_setresource with defined
Everything works well until I update some files inside
custom_data of Virtual machine scale set (vmss). Terraform appears to update
custom_data but as I use it as cloud-init it can’t be employed the second time.
Terraform, although it takes into account the changes in the
custom_data field, which we use for cloud-init provisioning, the changes are not applied since
cloud-init is processed only at the first boot.
I’ve seen that this issue https://github.com/terraform-providers/terraform-provider-azurerm/issues/490 led to changing the behaviour of
custom_data that was
ForceNew: true earlier. Moreover, it might have helped me if I were using that version but I don’t want to stick to it.
Instead, I wanted to use built-in provisioners for those purposes, for example remote-exec, but here another problem arises, we use DevOps, and for
remote-exec ssh connection is mandatory, which means you need to use the keys of those workers which deploy the Terraform module.
I see some problems here:
- there is a security issue that some hosts (Azure agents) is provided with access to production environment
- even if this was suitable option, then we’d need to understand how we can generally get the public key of the agent in the Terraform code itself to secure it in vmss afterwards
- we may use
local-exec, which could execute an Ansible playbook that requires ssh keys too, and even more (we don’t have IP addresses in attributes of any created resources) No DataSource is provide this information based on an ID or name of vmss. Thereafter, there would be a problem to make this pipe from terraform to ansible playbook though.
- we may install puppet/chef/salt right in the same Terraform module and through one of these tools ensure a state.
I cannot figure out how to apply these changes in a configuration management cycle. Of course, for each packer change, we can launch and build a new image with the required config, but this is contrary to the normal workflow.
To clarify I have a will to update Nginx configuration files and SSL certificates and reload the service on every update solely in Terraform module.
- bake an image on every change of configuration
- either update existing VM instances with the configs or create new ones
- use only Terraform for these purposes or if use configuration management to get IP addresses of VM instances deployed under that VMSS in the output of Terraform provider
- report IP addresses of current instances without looking in UI in order to automate everything
I would really love to hear how it works in your company!
Terraform + DevOps CI/CD pipeline without Kubernetes or script crutches.