Azurerm_storage_account doesn't properly disable CMK's & assigned identities

Terraform Providers Azure
1

Relatively interesting scenario I encountered while working through a small project to clean up some policy findings with my Azure resources. I had been tasked with discovering all of the Storage Accounts in our Azure Tenant that did not have Customer Managed Keys (CMKs) enabled and subsequently enabling CMKs for that account. Here’s the kicker though. An indeterminate amount of the storage accounts in my tenant were created by, and are managed via Terraform. Another indeterminate set of storage accounts were created manually outside of Terraform and are un-managed.

My plan of action to solve this issue was to write a script that would generate a unique system assigned identity for each storage account that did not have CMKs enabled, create a CMK key within a specified keyvault, and then enable CMK’s on the storage account using the key generated in the previous step.

Since some of our storage accounts are managed via Terraform I wanted to test the potential scenario where manually I enabled system assigned managed IDs and CMKs via a script, and then an unsuspecting developer executed Terraform plan/apply against the targeted resources I had previously modified outside of Terraform.

What I discovered is fairly interesting. It appears that when Terraform apply is executed against the resources I modified manually, Terraform will remove the managed identity from the storage account before trying to disable the CMK on the account. Removing the managed identity from the account first will prevent you from removing the CMK from the account. If this order of operations is followed the result is that the storage account is left in a “mangled” state where it is no longer possible to either assign another identity to the storage account, or remove the assigned CMK from the account.

I realize this is a bit of an edge case scenario. However, I figured it would be worth bringing to the community in hopes of getting the order of operations performed when Terraform disables managed ID’s and CMKs on a storage account updated.

For what it is worth, I was able to reproduce this issue outside of Terraform entirely following the same process using AZ CLI. I’ve opened a case with Microsoft in order to see why the affected storage account enters a “mangled” state when the identity is removed before the CMK.

image
image1079×703 215 KB

image
image
image
image1735×778 59.1 KB

image
image1263×245 121 KB

If you would like to reproduce this for yourself I’ve attached two powershell scripts below. The script assumes that the keyvault you’re using to store the CMK for the storage account is in the same resource group as the storage account.

You’ll also need to save the key rotation policy to a file in the same directory as the script.

Both scripts behave the same but perform the removal the CMK and assigned identity in different orders.

1 - Remove identity before removing CMK. Results in errors when trying to either assign another managed identity or remove the CMK.

2 - Remove the CMK prior to removing the identity. Works as intended and you are able to re-assign a new identity / CMK.

{
  "lifetimeActions": [
    {
      "trigger": {
        "timeAfterCreate": "P1Y",
        "timeBeforeExpiry": null
      },
      "action": {
        "type": "Rotate"
      }
    },
    {
      "trigger": {
        "timeBeforeExpiry": "P30D"
      },
      "action": {
        "type": "Notify"
      }
    }
  ],
  "attributes": {
    "expiryTime": "P2Y"
  }
}

$keyVaultRotationPolicyPath = "Path_To_Policy.json"

$saName = "NameOfYourStorageAccount"
$saRG = "NameOfYourResourceGroup"
$saManagedID = $null
$kvURL = "https://YourKeyVaultHere.vault.azure.net/"
$kvName = $null 
$kvNameRegex = "(?<=https:\/\/)(?<kvName>.*)(?=.vault)"
$keyName = $null


<#
    Scenario 1: Removing the system assigned managed ID from the storage account before removing the CMK attached to the account.
    Result: Subsequently unable to remove the storage account CMK (BadRequest)
            Unable to generate new system assigned managed ID (Code:missingMsiHeaders)
#>

Write-Host -ForegroundColor Green "Info: Begining Scenario 1 - Removing Identity before Removing the CMK"

#Create the account and store the managed ID
$saManagedID = az storage account create --name $saName --resource-group $saRG --location "eastus" --allow-blob-public-access "false" --allow-shared-key-access "false" --default-action "Deny" --bypass "AzureServices" --encryption-key-source "Microsoft.Storage" --https-only "true" --min-tls-version "TLS1_2" --public-network-access "enabled" --tags usecase="scriptTesting" --identity-type "SystemAssigned" --assign-identity --query "identity.principalId" --output tsv

#Build Keyvault Details
$kvURL -match $kvNameregex
$kvName = $matches.kvName

#Generate key
$randomString = -join ((65..90) + (97..122) | Get-Random -Count 5 | % {[char]$_})
$keyName = $saName + "-CMK-" + $randomString
az keyvault key create --vault-name $kvName --name $keyName --protection "software" --kty "RSA" --size "2048" --query "kid"

# Set rotation policy on the key
az keyvault key rotation-policy update --vault-name $kvName --name $keyName --value $keyVaultRotationPolicyPath

# Set an access policy on the key
az keyvault set-policy --name $kvName --resource-group $saRG --object-id $saManagedID --key-permissions "get" "unwrapKey" "wrapKey" "encrypt" "decrypt"

# Enable CMK on the storage account
Write-Host -ForegroundColor Yellow "Info: Scenario 1 - Enabling CMK on Storage Account"
az storage account update --name $saName --resource-group $saRG --encryption-key-source "Microsoft.Keyvault" --encryption-key-vault $kvURL --encryption-key-name $keyName --encryption-key-version ""

#Disable the system assigned managed id on the account
Write-Host -ForegroundColor Yellow "Info: Scenario 1 - Deleting the System Assigned Managed ID from the storage account"
az storage account update --name $saName --resource-group $saRG --identity-type "None"

# Try to Disable the CMK on the account () 
Write-Host -ForegroundColor Yellow "Info: Scenario 1 - Trying to disable the CMK on the storage account"
az storage account update --name $saName --resource-group $saRG --encryption-key-source "Microsoft.Storage"

# Try to re-generate a system assigned managed id on the account
Write-Host -ForegroundColor Yellow "Info: Scenario 1 - Trying to regenerate a system assigned managed ID the storage account"
az storage account update --name $saName --resource-group $saRG --identity-type "SystemAssigned" --assign-identity

# Delete the account
Write-Host -ForegroundColor Green "Info: Scenario 1 - Complete, Deleting Storage Account `n"
az storage account delete --name $saName --resource-group $saRG --yes

$keyVaultRotationPolicyPath = "Path_To_Policy.json"

$saName = "NameOfYourStorageAccount"
$saRG = "NameOfYourResourceGroup"
$saManagedID = $null
$kvURL = "https://YourKeyVaultHere.vault.azure.net/"
$kvName = $null 
$kvNameRegex = "(?<=https:\/\/)(?<kvName>.*)(?=.vault)"
$keyName = $null

Write-Host -ForegroundColor Green "Info: Begining Scenario 2 - Removing CMK before removing the identity"

#Create the account and store the managed ID
$saManagedID = az storage account create --name $saName --resource-group $saRG --location "eastus" --allow-blob-public-access "false" --allow-shared-key-access "false" --default-action "Deny" --bypass "AzureServices" --encryption-key-source "Microsoft.Storage" --https-only "true" --min-tls-version "TLS1_2" --public-network-access "enabled" --tags usecase="scriptTesting" --identity-type "SystemAssigned" --assign-identity --query "identity.principalId" --output tsv

#Build Keyvault Details
$kvURL -match $kvNameregex
$kvName = $matches.kvName

#Generate key
$randomString = -join ((65..90) + (97..122) | Get-Random -Count 5 | % {[char]$_})
$keyName = $saName + "-CMK-" + $randomString
az keyvault key create --vault-name $kvName --name $keyName --protection "software" --kty "RSA" --size "2048" --query "kid"

# Set rotation policy on the key
az keyvault key rotation-policy update --vault-name $kvName --name $keyName --value $keyVaultRotationPolicyPath

# Set an access policy on the key
az keyvault set-policy --name $kvName --resource-group $saRG --object-id $saManagedID --key-permissions "get" "unwrapKey" "wrapKey" "encrypt" "decrypt"

# Enable CMK on the storage account
Write-Host -ForegroundColor Yellow "Info: Scenario 2 - Enabling CMK on Storage Account"
az storage account update --name $saName --resource-group $saRG --encryption-key-source "Microsoft.Keyvault" --encryption-key-vault $kvURL --encryption-key-name $keyName --encryption-key-version ""

# Disable the CMK on the account () 
Write-Host -ForegroundColor Yellow "Info: Scenario 2 - Disabling the CMK on the storage account"
az storage account update --name $saName --resource-group $saRG --encryption-key-source "Microsoft.Storage"

# Disable the system assigned managed id on the account
Write-Host -ForegroundColor Yellow "Info: Scenario 2 - Deleting the System Assigned Managed ID from the storage account"
az storage account update --name $saName --resource-group $saRG --identity-type "None"

# Re-generate a system assigned managed id on the account
Write-Host -ForegroundColor Yellow "Info: Scenario 2 - Regenerating a system assigned managed ID to the storage account"
az storage account update --name $saName --resource-group $saRG --identity-type "SystemAssigned" --assign-identity

# Delete the account
Write-Host -ForegroundColor Green "Info: Scenario 2 - Complete, Deleting Storage Account. `n"
az storage account delete --name $saName --resource-group $saRG --yes