Summary
When changes are being made to an Azure Function App instance (with regional Vnet integration via subnet connected to App Service Plan), even if the changes do not include any networking-related stuff, the terraform apply step still fails, if the identity being used to run the command doesn’t have network change access.
I wonder if anyone has encountered similar issues before, and might be kind enough to share how the problem was worked around.
Details
Scenario
The scenario, in general, is as follows:
- Terraform is being run on a CI/CD agent
- The agent is Azure Pipelines self-hosted agent on a Windows VM
- Terraform uses the VM’s managed identity, which does not have permissions to mutate networks and will not get it
- The resource to manage is an Azure Function App
- Created with Terraform
- The Vnet integration is manually done after resource provisioned (CI/CD not applicable)
- Integrated to a subnet, specified in the App Service Plan
- After Vnet integrated, changes to the FuncApp are attempted
terraform plancorrectly shows the changes as it is in the code- most importantly, no network changes are involved
terraform applystill shows the right changes, but fails when applying them
The provider ver. is 3.93.0. Re-creating the resources from scratch (except the network) does not help.
Debug details
Using debug output, it was found that the interfering attribute is virtual_network_subnet_id, which has been ignored in the Terraform code. The tfstate keeps this attribute empty; when Terraform tries to connect to the AzureRM API endpoint, the request body correctly contains the actual non-empty subnet ID (which it obtained from prior calls to check the actual infrastructure state). This is where it fails inexplicably - AzureRM assumes a network change is being made where in reality there is none.
Terraform code snippet
resource "azurerm_linux_function_app" "dap_funcapp" {
name = "xxxxxxx"
resource_group_name = data.azurerm_resource_group.dap_rg.name
location = data.azurerm_resource_group.dap_rg.location
storage_account_name = data.azurerm_storage_account.dap_storage_acct.name
storage_account_access_key = data.azurerm_storage_account.dap_storage_acct.primary_access_key
service_plan_id = data.azurerm_service_plan.funcapp_service_plan.id
tags = local.tags
site_config {
always_on = true
app_service_logs {
disk_quota_mb = 35
retention_period_days = 0
}
application_stack {
python_version = var.python_runtime_version
}
}
app_settings = {
WEBSITES_ENABLE_APP_SERVICE_STORAGE = true
WEBSITE_ENABLE_SYNC_UPDATE_SITE = true
FUNCTIONS_WORKER_RUNTIME = "python"
FUNCTIONS_EXTENSION_VERSION = "~4"
SCM_DO_BUILD_DURING_DEPLOYMENT = false
AzureWebJobsFeatureFlags = "EnableWorkerIndexing"
BUILD_FLAGS = "UseExpressBuild"
ENABLE_ORYX_BUILD = "true"
XDG_CACHE_HOME = "/tmp/.cache"
}
lifecycle {
ignore_changes = [
virtual_network_subnet_id, site_config["ip_restriction"], site_config["vnet_route_all_enabled"]
]
}
}
Outputs
Full terraform apply output (note some sensitive information is redacted):
data.azurerm_resource_group.dap_rg: Reading...
data.azurerm_resource_group.dap_rg: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/redacted]
data.azurerm_service_plan.funcapp_service_plan: Reading...
data.azurerm_storage_account.dap_storage_acct: Reading...
data.azurerm_service_plan.funcapp_service_plan: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/serverFarms/redacted]
data.azurerm_storage_account.dap_storage_acct: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Storage/storageAccounts/redacted]
azurerm_linux_function_app.dap_funcapp: Refreshing state... [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx]
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# azurerm_linux_function_app.dap_funcapp will be updated in-place
~ resource "azurerm_linux_function_app" "dap_funcapp" {
~ app_settings = {
+ "FUNCTIONS_EXTENSION_VERSION" = "~4"
# (8 unchanged elements hidden)
}
id = "/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxxx"
name = "xxxxxxx"
# (27 unchanged attributes hidden)
# (1 unchanged block hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
azurerm_linux_function_app.dap_funcapp: Modifying... [id=/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx]
╷
│ Error: updating Linux App Service (Subscription: "redacted"
│ Resource Group Name: "redacted"
│ Site Name: "xxxxxxx"): performing CreateOrUpdate: unexpected status 403 with error: LinkedAuthorizationFailed: The client 'redacted' with object id 'redacted' has permission to perform action 'Microsoft.Web/sites/write' on scope '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx'; however, it does not have permission to perform action(s) 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy' (respectively) or the linked scope(s) are invalid.
│
│ with azurerm_linux_function_app.dap_funcapp,
│ on main.tf line 51, in resource "azurerm_linux_function_app" "dap_funcapp":
│ 51: resource "azurerm_linux_function_app" "dap_funcapp" {
│
│ updating Linux App Service (Subscription:
│ "redacted"
│ Resource Group Name: "redacted"
│ Site Name: "xxxxxxx"): performing CreateOrUpdate: unexpected
│ status 403 with error: LinkedAuthorizationFailed: The client
│ 'redacted' with object id
│ 'redacted' has permission to perform action
│ 'Microsoft.Web/sites/write' on scope
│ '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx';
│ however, it does not have permission to perform action(s)
│ 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked
│ scope(s)
│ '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy'
│ (respectively) or the linked scope(s) are invalid.
╵
##[error]Cmd.exe exited with code '1'.
Excerpt from the verbose debug output:
PUT /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx?api-version=2023-01-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 webapps/2023-01-01) HashiCorp Terraform/1.6.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.93.0 VSTS_664bc712-e7c8-4475-8f5b-9a88f9d8df97_build_250_0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 3911
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: redacted
Accept-Encoding: gzip
<request body. note: truncated>
{"id":"/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx","kind":"functionapp,linux","location": ... "virtualNetworkSubnetId":"/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy","vnetContentShareEnabled":false,"vnetImagePullEnabled":false,"vnetRouteAllEnabled":true},"tags": ... }
2024-04-11T11:36:44.437+0800 [DEBUG] provider.terraform-provider-azurerm_v3.93.0_x5.exe: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx?api-version=2023-01-01: timestamp="2024-04-11T11:36:44.423+0800"
2024-04-11T11:36:44.670+0800 [DEBUG] provider.terraform-provider-azurerm_v3.93.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx?api-version=2023-01-01:
HTTP/2.0 403 Forbidden
Content-Length: 767
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Apr 2024 03:36:44 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: redacted
X-Ms-Failure-Cause: gateway
X-Ms-Request-Id: redacted
X-Ms-Routing-Request-Id: KOREACENTRAL:20240411T033644Z:e4e75263-8d93-43af-9365-37ebc847a9aa
X-Msedge-Ref: Ref A: 8EBCB52744924EC88FD8A1146739450A Ref B: SEL221051504037 Ref C: 2024-04-11T03:36:44Z
{"error":{"code":"LinkedAuthorizationFailed","message":"The client 'redacted' with object id 'redacted' has permission to perform action 'Microsoft.Web/sites/write' on scope '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/xxxxxxx'; however, it does not have permission to perform action(s) 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/yyyyyyy' (respectively) or the linked scope(s) are invalid."}}: timestamp="2024-04-11T11:36:44.661+0800"
Any help would be deeply appreciated.