[Azure] Terraform refresh stops working after adding private endpoint to storage account

Terraform Providers Azure
1

I want to create an Azure storage account with restricted access, that is, the only IPs allowed to connects to it are my current public IP and some private IP from a given subnet. To ensure the traffic between the subnet and the storage account remains private, I want to use Azure private endpoint.

The Terraform config shown below works fine. My problem is not with the creation of the resources. My problem is once I add the private endpoint for the storage account, both ‘terraform refresh’ and ‘terraform plan’ commands stop working, so no further changes can be applied. Looking at the debug logs it seems there is some kind of infinite loop since the same group of messages can be seen repeating all over until a time-out error occurs. If I remove the private endpoint from the configuration, ‘terraform refresh’ and ‘terraform plan’ return to work as normal.

‘terraform refresh’ error message (with debug enabled is more than 1000 lines so not posting it here)

azurerm_resource_group.storage: Refreshing state... [id=/subscriptions/redacted-subscription-id/resourceGroups/test-storage-rg]
azurerm_resource_group.network: Refreshing state... [id=/subscriptions/redacted-subscription-id/resourceGroups/test-network-rg]
azurerm_virtual_network.vnet: Refreshing state... [id=/subscriptions/redacted-subscription-id/resourceGroups/test-network-rg/providers/Microsoft.Network/virtualNetworks/test-vnet]
azurerm_subnet.subnets["storage_account"]: Refreshing state... [id=/subscriptions/redacted-subscription-id/resourceGroups/test-network-rg/providers/Microsoft.Network/virtualNetworks/test-vnet/subnets/storage-account-snet]
azurerm_subnet.subnets["web"]: Refreshing state... [id=/subscriptions/redacted-subscription-id/resourceGroups/test-network-rg/providers/Microsoft.Network/virtualNetworks/test-vnet/subnets/web-snet]
module.test_sa.azurerm_storage_account.this: Refreshing state... [id=/subscriptions/redacted-subscription-id/resourceGroups/test-storage-rg/providers/Microsoft.Storage/storageAccounts/testrandomname]

Error: retrieving static website properties for Storage Account (Subscription: "redacted-subscription-id"
Resource Group Name: "test-storage-rg"
Storage Account Name: "testrandomname"): accounts.Client#GetServiceProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

  with module.test_sa.azurerm_storage_account.this,
  on modules/storage_account/main.tf line 12, in resource "azurerm_storage_account" "this":
  12: resource "azurerm_storage_account" "this" {

Root module

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.84.0"
    }
  }

  required_version = ">= v1.5.7"
}

provider "azurerm" {
  subscription_id = "redacted"
  tenant_id       = "redacted"
  features {}
}

## Network ####################################################################################################################################

resource "azurerm_resource_group" "network" {
  location = "eastasia"
  name     = "test-network-rg"
}

resource "azurerm_virtual_network" "vnet" {
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.network.location
  name                = "test-vnet"
  resource_group_name = azurerm_resource_group.network.name
}

resource "azurerm_subnet" "subnets" {
  for_each = {
    web = {
      prefixes          = ["10.0.2.0/24"]
      service_endpoints = ["Microsoft.Web", "Microsoft.Storage"]
    }
    storage_account = {
      prefixes          = ["10.0.3.0/24"]
      service_endpoints = ["Microsoft.Storage"]
    }
  }

  address_prefixes     = each.value.prefixes
  name                 = format("%s-snet", replace(each.key, "_", "-"))
  resource_group_name  = azurerm_resource_group.network.name
  service_endpoints    = each.value.service_endpoints
  virtual_network_name = azurerm_virtual_network.vnet.name
}

## Storage ####################################################################################################################################

resource "azurerm_resource_group" "storage" {
  location = "eastasia"
  name     = "test-storage-rg"
}

data "http" "my_public_ip" {
  url = "http://ipecho.net/plain"
}

module "test_sa" {
  allowed_ips                = [data.http.my_public_ip.body]
  allowed_subnet_ids         = [azurerm_subnet.subnets["web"].id]
  location                   = azurerm_resource_group.storage.location
  name                       = replace("test-random-name", "-", "")
  private_endpoint_name      = "test-sa-%s-pep"
  private_endpoint_subnet_id = azurerm_subnet.subnets["storage_account"].id
  resource_group_name        = azurerm_resource_group.storage.name
  source                     = "./modules/storage_account"
}

Contents of modules/storage_account/main.tf

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.84.0"
    }
  }

  required_version = ">= v1.5.7"
}

resource "azurerm_storage_account" "this" {
  access_tier                     = var.access_tier
  account_kind                    = var.account_kind
  account_replication_type        = var.account_replication_type
  account_tier                    = var.account_tier
  allow_nested_items_to_be_public = false
  location                        = var.location
  name                            = var.name
  public_network_access_enabled   = length(var.allowed_ips) > 0
  resource_group_name             = var.resource_group_name

  network_rules {
    bypass                     = ["AzureServices", "Logging", "Metrics"]
    default_action             = "Deny"
    ip_rules                   = var.allowed_ips
    virtual_network_subnet_ids = var.allowed_subnet_ids
  }
}

# This is the issue. Once added both `terraform refresh` and `terraform plan` stop working
resource "azurerm_private_endpoint" "this" {
  for_each                      = var.private_endpoint_subresources
  custom_network_interface_name = format("%s-nic", format(var.private_endpoint_name, each.value))
  location                      = var.location
  name                          = format(var.private_endpoint_name, each.value)
  resource_group_name           = var.resource_group_name
  subnet_id                     = var.private_endpoint_subnet_id

  private_service_connection {
    is_manual_connection           = false
    name                           = format(var.private_endpoint_name, each.value)
    private_connection_resource_id = azurerm_storage_account.this.id
    subresource_names              = [each.value] # Storage Account only supports one subresource a time, hence the for_each loop
  }
}

Contents of modules/storage_account/variables.tf

variable "access_tier" {
  default     = "Hot"
  description = "Default access tier to use for the Storage Account."
  type        = string
}

variable "account_kind" {
  default     = "StorageV2"
  description = "Kind of Storage Account."
  type        = string
}

variable "account_replication_type" {
  default     = "LRS"
  description = "Type of replication to use for the Storage Account."
  type        = string
}

variable "account_tier" {
  default     = "Standard"
  description = "Tier to use for the Storage Account."
  type        = string
}

variable "allowed_ips" {
  default     = []
  description = "Public IPs allowed to access the Storage Account."
  type        = list(string)
}

variable "allowed_subnet_ids" {
  description = "IDs of the Subnets which should be able to access the Storage Account."
  type        = list(string)
}

variable "location" {
  description = "Azure region where resources created by this module will reside in."
  type        = string
}

variable "name" {
  description = "Name of the created Storage Account."
  type        = string
}

variable "private_endpoint_name" {
  description = "Template to generate the name of the created Private Endpoints."
  type        = string
  validation {
    condition     = length(regexall("%s", var.private_endpoint_name)) == 1
    error_message = "The name must contain '%s' as a placeholder that will be replaced with the Storage Account subresource name."
  }
}

variable "private_endpoint_subnet_id" {
  description = "ID of the subnet from which the Private Endpoint IPs will be allocated."
  type        = string
}

variable "private_endpoint_subresources" {
  default     = ["blob"]
  description = "List of Storage Account subresource types (Blob, Files, Tables, Queues) for which to create a Private Endpoint"
  type        = set(string)
}

variable "resource_group_name" {
  description = "Name of the Resource Group where resources created by this module will reside in."
  type        = string
}
2

Hi there:

We are running into the same issue here. Any solution yet? I’ll add anything that I come up with on this post.

Thanks

3

where do you run terraform from? your own laptop or a pipeline?

4

To resolve the issue with terraform refresh and terraform plan commands stopping after adding a private endpoint to an Azure storage account, ensure the DNS settings are correctly configured for private access, update to the latest AzureRM provider version, verify network policies, and check the Terraform configuration for errors. If these steps don’t help, consider temporarily removing the private endpoint configuration to proceed with other updates and then re-adding it.

5

If you cannot fix the issue I can find it in my [Azure RAG]

Let me know

6

This is by default. To solve this problem, run your terraform from a machine that has access to the network in which the private endpoint is deployed.

The reason is terraform will try to refresh storage endpoints via its fqdn and not true the resource ID. Because you enable the PE, the fqdn public endpoint is inaccessible and can only remove the the fqdn PE via a network connected machine.

Let me know if this helps.