Update VMSS (custom_data of instances) via Azure DevOps

Terraform
1

Hey guys!

What is the correct way to deploy and upgrade virtual machines in the Azure Cloud using Terraform Azure Provider?

  • I use Packer to bake an image with all required linux packages. This step is executed in the first job of Release pipeline in Azure DevOps (with Azure-provided agents).
  • On the next task I deploy the entire infrastructure via Terraform where the key point is usage of azurerm_linux_virtual_machine_scale_set resource with defined custom_data inside which #cloud-config with write_files and runcmd.

Everything works well until I update some files inside custom_data of Virtual machine scale set (vmss). Terraform appears to update custom_data but as I use it as cloud-init it can’t be employed the second time.

Terraform, although it takes into account the changes in the custom_data field, which we use for cloud-init provisioning, the changes are not applied since cloud-init is processed only at the first boot.
I’ve seen that this issue https://github.com/terraform-providers/terraform-provider-azurerm/issues/490 led to changing the behaviour of custom_data that was ForceNew: true earlier. Moreover, it might have helped me if I were using that version but I don’t want to stick to it.

Instead, I wanted to use built-in provisioners for those purposes, for example remote-exec, but here another problem arises, we use DevOps, and for remote-exec ssh connection is mandatory, which means you need to use the keys of those workers which deploy the Terraform module.
I see some problems here:

  • there is a security issue that some hosts (Azure agents) is provided with access to production environment
  • even if this was suitable option, then we’d need to understand how we can generally get the public key of the agent in the Terraform code itself to secure it in vmss afterwards
  • we may use local-exec, which could execute an Ansible playbook that requires ssh keys too, and even more (we don’t have IP addresses in attributes of any created resources) No DataSource is provide this information based on an ID or name of vmss. Thereafter, there would be a problem to make this pipe from terraform to ansible playbook though.
  • we may install puppet/chef/salt right in the same Terraform module and through one of these tools ensure a state.

I cannot figure out how to apply these changes in a configuration management cycle. Of course, for each packer change, we can launch and build a new image with the required config, but this is contrary to the normal workflow.

To clarify I have a will to update Nginx configuration files and SSL certificates and reload the service on every update solely in Terraform module.

Don’t want:

  • bake an image on every change of configuration

Want:

  • either update existing VM instances with the configs or create new ones
  • use only Terraform for these purposes or if use configuration management to get IP addresses of VM instances deployed under that VMSS in the output of Terraform provider
  • report IP addresses of current instances without looking in UI in order to automate everything

I would really love to hear how it works in your company!
Terraform + DevOps CI/CD pipeline without Kubernetes or script crutches.

Thank you!

2

Cool, I found an undocumented beta feature: this extension block for vmss is analogous to the azurerm_virtual_machine_extension resource, where you can set a custom script that will always be executed upon update. But for this feature to work, you need to check here for the existence of this key in map like this

if val, ok: = extensionRaw ["protected_settings"]; ok {
protectedSettings, err: = structure.ExpandJsonFromString (val. (string))
if err! = nil {
return nil, fmt.Errorf ("failed to parse JSON from` settings`:% + v ", err)
}
extensionProps.ProtectedSettings = protectedSettings
}

otherwise, you’ll get a JSON unmarshalling problem as you pass a blank string.
Then compile the binary and run it with an environment variable:
ARM_PROVIDER_VMSS_EXTENSIONS_BETA = true
Profit. On each update, the resource is not recreated, but the script is executed. It remains to write a bash, wrap the file configuration in it and wait for the release of this version.