Unable to update GCP Security Policy - HTTP references are not supported for security policies with deny rules

We have a number of security policies which are basically whitelists of IP addresses.

These have been working fine for awhile, we update them via pipelines called from Jenkins which run the terraform for us, so I can’t easily run the terraform directly to debug things, but there a few things I can try if needs be.

The problem we’ve had is we’ve gone to add a new IP range to our WAF, and we’re suddenly getting an error when adding the range:

Error: Error updating RegionSecurityPolicy "projects/project_id/regions/europe-west1/securityPolicies/customer-security-policy": googleapi: Error 400: HTTP references are not supported for security policies with deny rules., badRequest

At first search, it appeared this was due to the default deny rule returning a 403 however I reset this default rule to simply return deny instead of deny(403) and still we see this error each time we attempt to add the new IP range to a allow rule.

The plan is only showing the change to the allow rule, and only to add or remove (i’ve tried both) a IP range as the only value that’s changing:

# google_compute_region_security_policy.cloud_armor_policies["customer"] will be updated in-place
~ resource "google_compute_region_security_policy" "cloud_armor_policies" {
id = "projects/project_id/regions/europe-west1/securityPolicies/customer-security-policy"
name = "customer-security-policy"
# (7 unchanged attributes hidden)

~ rules {
# (4 unchanged attributes hidden)

~ match {
# (1 unchanged attribute hidden)

~ config {
~ src_ip_ranges = [
# (3 unchanged elements hidden)
"10.1.37.160/27",
- "10.1.0.1/32",
]
}
}
}

# (5 unchanged blocks hidden)
}

Considering this is the only change to the policy, the error message given doesn’t seem to make sense and I wonder if in fact there’s a different error occuring?
I’m about 90% sure we’ve added/removed IP ranges on this version of our code, and that we’ve not introduced some bug through updates.

Also, manually updating the rule, either via the console or via a gcloud command works successfully, adding and removing the networks as required, it’s only when called via the terraform that this is failing.

I don’t believe in this case it’s relevant, but I’m adding the code we’re using for generating these security policies here:

resource "google_compute_region_security_policy" "cloud_armor_policies" {
  for_each   = local.security_policies
  name       = "${each.key}-security-policy"
  project    = local.project
  description = "basic region security for ${each.key} loadbalancer"
  type       = "CLOUD_ARMOR"
  region = lookup({
    "region1" = local.region1
    "region2" = local.region2
    "region3" = local.region3
    "region4" = local.region4
  }, each.value.region, local.region2)

  dynamic "rules" {
    for_each = flatten([
      for rule in [
        for p in sort([
          for r in each.value.rules : tostring(r.priority)
        ]) : (
          # Find matching rule with this priority
          one([
            for r in each.value.rules : {
              priority       = tonumber(r.priority)
              action         = r.action
              versioned_expr = try(r.versioned_expr, null)
              expression     = try(r.expression, null)
              description    = try(r.description, "no description given")
              src_ip_ranges  = try(r.src_ip_ranges, null)
            } if tostring(r.priority) == p
          ])
        )
      ] : (
        length(coalesce(rule.src_ip_ranges, [])) > 0 ?
          [
            for chunk_index, ip_chunk in chunklist(flatten([
              for ip in rule.src_ip_ranges :
                can(local.resolved_placeholders[trim(ip, "{}")])
                  ? local.resolved_placeholders[trim(ip, "{}")]
                  : [ip]
            ]), 10) : {
              action         = rule.action
              base_priority  = rule.priority
              chunk_index    = chunk_index
              description    = rule.description
              versioned_expr = rule.versioned_expr
              expression     = rule.expression
              src_ip_ranges  = ip_chunk
            }
          ]
        :
          [{
            action         = rule.action
            base_priority  = rule.priority
            chunk_index    = 0
            description    = rule.description
            versioned_expr = null
            expression     = rule.expression
            src_ip_ranges  = null
          }]
      )
    ])

    content {
      action      = rules.value.action
      priority    = rules.value.base_priority + rules.value.chunk_index
      description = "${rules.value.description} (chunk ${rules.value.chunk_index + 1})"

      match {
        dynamic "expr" {
          for_each = rules.value.expression != null ? [1] : []
          content {
            expression = rules.value.expression
          }
        }

        versioned_expr = rules.value.versioned_expr

        dynamic "config" {
          for_each = rules.value.src_ip_ranges != null ? [1] : []
          content {
            src_ip_ranges = rules.value.src_ip_ranges
          }
        }

      }
    }
  }
}

And the config we pass in:

        "ingress-test-customer": {
            "region": "region2",
            "output_group": "test",
            "output_identity": "external",
            "rules": [
                {
                    "action": "allow",
                    "priority": "1001",
                    "versioned_expr": "SRC_IPS_V1",
                    "src_ip_ranges": ["{{customer_nonprod_subnets}}"],
                    "description": "default rule"
                },
                {
                    "action": "deny",
                    "priority": "2147483647",
                    "versioned_expr": "SRC_IPS_V1",
                    "src_ip_ranges": ["{{all_ips}}"],
                    "description": "default deny rule"
                }
            ]
        }

I’ve finally replicated this issue in my test project, simply by having a backend on the WAF as a target, and hey presto, can no longer update the WAF.

I remove the target, and I can once again successfully update the IP ranges in the WAF. I still don’t understand the error given, nor why this has started happening but I can only presume there’s been some change on the GCP side which means the way terraform was updating the WAF is no longer valid when there’s endpoints attached?

I’ve figured out the cause of the error.
We have existing rules, which are returning a deny, instead of deny(http_error_code). These rules are no longer accepted via terraform, or if you try and create them via the console (it forces you to set the error code).

However, if you create rules via the gcloud API, the deny rules with no error code are accepted, and display normally etc but when you then try to modify them, the error occurs. I’m unsure if this is a problem on the GCP side, the provider code or something else.