Terraform State Refresh Error with Azurerm Storage account with private endpoint

Terraform
2

Hi @ceciivanov,

This is a common issue with the way the azurerm provider has been written and handles storage accounts.

The main storage account itself is provisioned and configured by the provider using the Azure Resource Manager APIs (The Azure ‘control plane’), however the sub resources, such as containers are created by the azurerm provider accessing the storage account directly (not via the ARM API) so therefore requires access via the ‘data plane’ as soon as you want to manipulate these sub resources.

This means that (if you are provisioning these sub resources) the device that you are running your terraform plan/apply on requires access to the storage account itself. Of course, this can be a problem when the security policy that needs to be applied to the storage account mandates that it must not be publicly accessible (e.g… use of Private endpoints or service endpoints only) or that any public access to the storage account must be controlled by the storage account firewall (IP whitelisting).

Further issues arise if you are using pipeline agents (e.g… Azure DevOps pipelines) to provision your infrastructure:

  • If you are using ‘Microsoft Hosted’ pipeline agents then they will need access to the storage account via the public network. Furthermore, the pipeline agents can run in any of the regions in the geography (e.g… If you have your devops org geo set to Europe then the agents can be instantiated in either the North Europe or West Europe region. That’s a lot of Azure IP addresses to whitelist on a storage account and makes ‘securing’ the Storage account a bit of a joke.

  • If you are using Self-Hosted pipeline agents then you need to ensure that the agent has access to the data plane. This could be as straightforward as IP whitelisting an IP address that all of your org’s traffic to the internet originates from. But more often that not, it means ensuring the agent has the ability to route through your private network to hit the storage accounts private endpoint or that it is in a subnet with with storage service endpoints provisioned (and you have the relevant subnet allowed in the storage firewall)

If you are not able to go down the self-hosted agent route, or for some reason have to run Terraform on a device that is outside your private network / can’t privately route there is an approach you can use to mitigate this and move the ‘subresource’ provisioning over to use the control-plane (removing the requirement to access the storage account directly by the agent:

Provision the storage account and all configuration using the AzureRM provider, but utilise the AzAPI provider (which ONLY uses the ARM API / data plane) to then provision and configure the sub resources similar to the below:

resource "azapi_resource" "containers" {

  type = "Microsoft.Storage/storageAccounts/blobServices/containers@2022-09-01"
  body = {
    properties = {
      publicAccess = false
    }
  }
  name                      = "my_container_name"
  parent_id                 = "${azurerm_storage_account.id}/blobServices/default"  # <-- Reference to the storage account deployed via AzureRM Providcer
  schema_validation_enabled = false 
    # https://github.com/Azure/terraform-provider-azapi/issues/497
}

For a more complete example see this from the Azure Verified module project’s storage account module.

Also you may need to refer to: Microsoft.Storage/storageAccounts/blobServices/containers - Bicep, ARM template & Terraform AzAPI reference | Microsoft Learn

Hope that helps

Happy Terraforming!

Note: in my code above I am using the ‘dynamic schema’ capability that is available from azapi provider release v1.13.0, whereas the AVM uses the older approach requiring a jsonencode() function for the body attribute