Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>

I have the following terraform plan

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # module.keyvault.azurerm_monitor_diagnostic_setting.kv will be created
+ resource "azurerm_monitor_diagnostic_setting" "kv" {
      + id                             = (known after apply)
      + log_analytics_destination_type = (known after apply)
      + log_analytics_workspace_id     = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/rg-euw-mgmt/providers/Microsoft.OperationalInsights/workspaces/log-euw-mgmt-shared"
      + name                           = "setByTerraform"
      + storage_account_id             = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/rg-euw-mgmt/providers/Microsoft.Storage/storageAccounts/steuwmgmtlog82398"
      + target_resource_id             = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/rg-aksdev-euw-mgmt/providers/Microsoft.KeyVault/vaults/kv-aksdev-euw"

      + enabled_log {
          + category_group = "allLogs"
        }
      + enabled_log {
          + category_group = "audit"
        }

      + log {
          + category       = (known after apply)
          + category_group = (known after apply)
          + enabled        = (known after apply)

          + retention_policy {
              + days    = (known after apply)
              + enabled = (known after apply)
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

But it fails with

module.keyvault.azurerm_monitor_diagnostic_setting.kv: Creating...
╷
│ Error: creating Monitor Diagnostics Setting "setByTerraform" for Resource "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/rg-aksdev-euw-mgmt/providers/Microsoft.KeyVault/vaults/kv-aksdev-euw": diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>
│ 
│   with module.keyvault.azurerm_monitor_diagnostic_setting.kv,
│   on .terraform/modules/keyvault/keyvault/main.tf line 56, in resource "azurerm_monitor_diagnostic_setting" "kv":
│   56: resource "azurerm_monitor_diagnostic_setting" "kv" {
│ 
╵

I am not sure why, and wonder how I can solve that problem.

I also found azurerm_monitor_diagnostic_setting fails to create with 409 nil error · Issue #21161 · hashicorp/terraform-provider-azurerm (github.com) but no solution there.

Have you checked the KeyVault diagnostic settings to make sure it’s not already configured?

I checked now, yes there are 2 settings defined. But then again, not the one we try to configure

resource "azurerm_monitor_diagnostic_setting" "kv" {
  name                       = "setByTerraform"
  target_resource_id         = azurerm_key_vault.kv.id
  log_analytics_workspace_id = var.centralLogAnalyticWorkspaceId
  storage_account_id         = var.centralLogAnalyticStorageAccountID

  dynamic "enabled_log" {
    for_each = data.azurerm_monitor_diagnostic_categories.kv.log_category_groups
    content {
      category_group = enabled_log.value
    }
  }
}

Looks like we had a Azure policy configured, which does configure the diagnostic settings, so we will remove the config in the aks setup to fix the error