[ARM Builder] Keyvault access denied to first party service [Azure Compute]

jayendran.arumugam · November 6, 2023, 6:48am

I have used the build_key_vault_name by brining my own keyvault which is having network restriction and allowed only to my private network within vnet however packer is somewhat using the Azure Compute to fetch the secret using a Public IP belongs to MS.

My Packer log

azure-arm: output will be in this color.
11:52:45  
11:52:45  ==> azure-arm: Running builder ...
11:52:45  ==> azure-arm: Getting tokens using client secret
11:52:45  ==> azure-arm: Getting tokens using client secret
11:52:45      azure-arm: Creating Azure Resource Manager (ARM) client ...
11:52:45  ==> azure-arm: Using existing resource group ...
11:52:45  ==> azure-arm:  -> ResourceGroupName : 'images-storage-rg-01'
11:52:45  ==> azure-arm:  -> Location          : 'eastus2'
11:52:45  ==> azure-arm: Setting the certificate in the KeyVault...
11:52:45  ==> azure-arm: Getting the certificate's URL ...
11:52:45  ==> azure-arm:  -> Key Vault Name        : 'packer-kv-01'
11:52:45  ==> azure-arm:  -> Key Vault Secret Name : 'packerKeyVaultSecret'
11:52:45  ==> azure-arm:  -> Certificate URL       : 'https://packer-kv-01.vault.azure.net/secrets/packerKeyVaultSecret/0899826f9b724a84af004756f9545236'
11:52:45  ==> azure-arm: Setting the certificate's URL ...
11:52:45  ==> azure-arm: Validating deployment template ...
11:52:45  ==> azure-arm:  -> ResourceGroupName : 'images-storage-rg-01'
11:52:45  ==> azure-arm:  -> DeploymentName    : 'pkrdpz90dt1tlc0'
11:52:45  ==> azure-arm: Deploying deployment template ...
11:52:45  ==> azure-arm:  -> ResourceGroupName : 'images-storage-rg-01'
11:52:45  ==> azure-arm:  -> DeploymentName    : 'pkrdpz90dt1tlc0'
11:52:45  ==> azure-arm: ERROR: -> DeploymentFailed : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
11:52:45  ==> azure-arm: ERROR:   -> Conflict
11:52:45  ==> azure-arm: ERROR:   -> ResourceDeploymentFailure : The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.
11:52:45  ==> azure-arm: ERROR:     -> KeyVaultAccessForbidden : Key Vault https://packer-kv-01.vault.azure.net/secrets/packerKeyVaultSecret/0899826f9b724a84af004756f9545236 either has not been enabled for deployment or the vault id provided, /subscriptions/****/resourceGroups/images-storage-rg-01/providers/Microsoft.KeyVault/vaults/packer-kv-01, does not match the Key Vault's true resource id.
11:52:45  ==> azure-arm:
11:52:45  ==> azure-arm: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"KeyVaultAccessForbidden\",\r\n        \"message\": \"Key Vault https://packer-kv-01.vault.azure.net/secrets/packerKeyVaultSecret/0899826f9b724a84af004756f9545236 either has not been enabled for deployment or the vault id provided, /subscriptions/****/resourceGroups/images-storage-rg-01/providers/Microsoft.KeyVault/vaults/packer-kv-01, does not match the Key Vault's true resource id.\"\r\n      }\r\n    ]\r\n  }\r\n}"}]
11:52:45  ==> azure-arm: 
11:52:45  ==> azure-arm: The resource group was not created by Packer, deleting individual resources ...
11:52:45  ==> azure-arm: Removing the created Deployment object: 'pkrdpz90dt1tlc0'
11:52:45  ==> azure-arm: 
11:52:45  ==> azure-arm: The resource group was not created by Packer, not deleting ...
11:52:45  Build 'azure-arm' errored after 1 minute 17 seconds: unexpected EOF

My Keyvault audit logs

Access denied to first party service.
Caller: name=Compute;tid=f8cdef31-a31e-4b4a-93e4-5f571e91255a;appid=579d9c9d-4c83-4efc-8124-7eba65ed3356;oid=9486e527-3706-4e7b-8295-613aac964938;iss=https://sts.windows.net/f8cdef31-a31e-4b4a-93e4-5f571e91255a/
Vault:packer-kv-01;location=eastus2

Public IP: 52.136.29.5
Operation: SecretGet

Question:

Why packer is using “Azure Compute” SPN to fetch the secret and not the SPN that I gave ?
Why the keyvault get secret operation is initiating from MS public IP and not the private IP ?

backerman · March 4, 2024, 1:45am

You’re building a VM with a secret to be retrieved from Key Vault (AKV) once at the time of image creation and stored in the system store. As such, CRP¹ retrieves the secret from Key Vault under its own identity; therefore, the key vault must be enabled for deployment (which it currently is not). If you look up that IP address in the service tags data ,you’ll find it in AzureCloud.eastus2.

Explanation: The Azure ARM builder uses a hardcoded template in template_builder.go; the references to the vault and secret name are added in the BuildWindows method towards the top of that file.

When deploying a VM to directly use, you’d normally want to deploy certificates with the AKV extension, which authenticates using the VM’s managed identity and does not require the vault to be enabled for deployment. I’m not sure why this method was selected; it’s possible that it took too long for the AKV extension to be provisioned, and since the certificate is removed in the deprovisioning process, it will never need to be updated.

(I am a Microsoft employee, but this is not an official support channel and I’m on vacation for the next day and a half, so insert any and all appropriate disclaimers here.)

¹ Compute Resource Provider