Dynamically generating alerts based on resources and alert definitions - approach validation

Terraform
1

Hi - My goal is to create a number of (metric) alerts for each type of resource I find within an Azure subscription based upon a pre-defined number of alert definitions for each resource type. For example, for each Storage Account found in a subscription I may want to configure one distinct alert and, for each VM found, two.

I have a module with something akin to the following (cut down) code, which works ‘ok’ currently, but is likely not the ‘best’ way to achieve that goal.

A map of alert definitions and what type of resource, based on its ‘metricnamespace’, it applies to (in this example, there are intended to be 2 for each VM and 1 for each Storage Account):

locals {
  metric_alert_defs = {
    alertdef1 = {
      criteriontype = "StaticThresholdCriterion"
      metricnamespace = "Microsoft.Compute/virtualMachines"
      severity = "2"
    }
    alertdef2 = {
      criteriontype = "StaticThresholdCriterion"
      metricnamespace = "Microsoft.Compute/virtualMachines"
      severity = "3"
    }
    alertdef3 = {
      criteriontype = "StaticThresholdCriterion"
      metricnamespace = "Microsoft.Storage/storageAccounts"
      severity = "4"
    }
  }
}

A set containing all of the VM and Storage Account resources found in the Subscription retrieved using the respective data resource:

locals {
  resources = toset(concat(data.azurerm_resources.sub_virtualMachines.resources[*],
    data.azurerm_resources.sub_storageAccounts.resources[*]))
}

The user of the module has the option of passing in an ‘overrides’ map also, metric_alert_definition_overrides - this is the same structure as the alerts definition map above and simply allows the user to ‘redefine’ one of more attributes of a pre-defined alert definition. For instance they may want to override the Severity of ‘alertdef3’ to be 1 in their Subscription (this would consequently affect the alert on every Storage Account where alertdef3 is applied)

metric_alerts_map_with_definition_overrides = {
    for key, value in local.metric_alerts_map : key => merge( value, lookup( var.metric_alert_definition_overrides, key, {} ) )
  }

With those two items defined and the ‘overrides’, if any, applied, I can now construct a map which would contain - for a Subscription with 2 VMs and 3 Storage Accounts - 7 elements (i.e. 2 alerts per VM and 1 alert per Storage Account):

resources_alerts = distinct(flatten([
  for r in local.resources : [
    for a in local.metric_alerts_map_with_definition_overrides : {
	alert_criteriontype     = a.criteriontype
        alert_severity              = a.severity
      }
      if a.metricnamespace == r.type
    ]
  ]))

These (7) alerts are then actually created using using:

resource "azurerm_monitor_metric_alert" "my_static_metric_alert_rules" {
  for_each = { for k, v in local.resource_alerts : k => v if v.alert_criteriontype == "StaticThresholdCriterion" }

  severity                 = each.value.alert_severity
.
.
}

While that code works, I’m pretty sure it’s less than optimal - for instance when there’s an additional to, or removal from, the ‘metric_alert_defs’ map, TF will usually want to destroy and then re-create a good number of the alerts.

Also, while the mechanism to override one or more attributes of an alert’s definition works fine, it isn’t granular enough to ‘tweak’ an alert for a specific resource. In an attempt to achieve that, my first thought was to defined an additional, more granular, ‘resourceoverride’ map (defining the ‘target’ resource name and the required attributes for its particular alert) and then use code similar to the one that defines metric_alerts_map_with_definition_overrides above to update the previously generated (7) resources_alerts - I couldn’t for the life of me get that to work though; it never applied any of these more granular updates, I suspect due to differences in the data structures.

I suspect there’s a far better mechanism to achieve this but it’s beyond my current TF knowledge. The main asks are:

  • How best to avoid the alert destroy / re-create on alert definition additions / removals?
  • How to account for the need to override on a per-resource basis if necessary?

thanks