We have a number of security policies which are basically whitelists of IP addresses.
These have been working fine for awhile, we update them via pipelines called from Jenkins which run the terraform for us, so I can’t easily run the terraform directly to debug things, but there a few things I can try if needs be.
The problem we’ve had is we’ve gone to add a new IP range to our WAF, and we’re suddenly getting an error when adding the range:
│ Error: Error updating RegionSecurityPolicy "projects/project_id/regions/europe-west1/securityPolicies/customer-security-policy": googleapi: Error 400: HTTP references are not supported for security policies with deny rules., badRequest
At first search, it appeared this was due to the default deny rule returning a 403 however I reset this default rule to simply return deny instead of deny(403) and still we see this error each time we attempt to add the new IP range to a allow rule.
The plan is only showing the change to the allow rule, and only to add or remove (i’ve tried both) a IP range as the only value that’s changing:
# google_compute_region_security_policy.cloud_armor_policies["customer"] will be updated in-place
~ resource "google_compute_region_security_policy" "cloud_armor_policies" {
id = "projects/project_id/regions/europe-west1/securityPolicies/customer-security-policy"
name = "customer-security-policy"
# (7 unchanged attributes hidden)
~ rules {
# (4 unchanged attributes hidden)
~ match {
# (1 unchanged attribute hidden)
~ config {
~ src_ip_ranges = [
# (3 unchanged elements hidden)
"10.1.37.160/27",
- "10.1.0.1/32",
]
}
}
}
# (5 unchanged blocks hidden)
}
Considering this is the only change to the policy, the error message given doesn’t seem to make sense and I wonder if in fact there’s a different error occuring?
I’m about 90% sure we’ve added/removed IP ranges on this version of our code, and that we’ve not introduced some bug through updates.
Also, manually updating the rule, either via the console or via a gcloud command works successfully, adding and removing the networks as required, it’s only when called via the terraform that this is failing.
I don’t believe in this case it’s relevant, but I’m adding the code we’re using for generating these security policies here:
resource "google_compute_region_security_policy" "cloud_armor_policies" {
for_each = local.security_policies
name = "${each.key}-security-policy"
project = local.project
description = "basic region security for ${each.key} loadbalancer"
type = "CLOUD_ARMOR"
region = lookup({
"region1" = local.region1
"region2" = local.region2
"region3" = local.region3
"region4" = local.region4
}, each.value.region, local.region2)
dynamic "rules" {
for_each = flatten([
for rule in [
for p in sort([
for r in each.value.rules : tostring(r.priority)
]) : (
# Find matching rule with this priority
one([
for r in each.value.rules : {
priority = tonumber(r.priority)
action = r.action
versioned_expr = try(r.versioned_expr, null)
expression = try(r.expression, null)
description = try(r.description, "no description given")
src_ip_ranges = try(r.src_ip_ranges, null)
} if tostring(r.priority) == p
])
)
] : (
length(coalesce(rule.src_ip_ranges, [])) > 0 ?
[
for chunk_index, ip_chunk in chunklist(flatten([
for ip in rule.src_ip_ranges :
can(local.resolved_placeholders[trim(ip, "{}")])
? local.resolved_placeholders[trim(ip, "{}")]
: [ip]
]), 10) : {
action = rule.action
base_priority = rule.priority
chunk_index = chunk_index
description = rule.description
versioned_expr = rule.versioned_expr
expression = rule.expression
src_ip_ranges = ip_chunk
}
]
:
[{
action = rule.action
base_priority = rule.priority
chunk_index = 0
description = rule.description
versioned_expr = null
expression = rule.expression
src_ip_ranges = null
}]
)
])
content {
action = rules.value.action
priority = rules.value.base_priority + rules.value.chunk_index
description = "${rules.value.description} (chunk ${rules.value.chunk_index + 1})"
match {
dynamic "expr" {
for_each = rules.value.expression != null ? [1] : []
content {
expression = rules.value.expression
}
}
versioned_expr = rules.value.versioned_expr
dynamic "config" {
for_each = rules.value.src_ip_ranges != null ? [1] : []
content {
src_ip_ranges = rules.value.src_ip_ranges
}
}
}
}
}
}
And the config we pass in:
"ingress-test-customer": {
"region": "region2",
"output_group": "test",
"output_identity": "external",
"rules": [
{
"action": "allow",
"priority": "1001",
"versioned_expr": "SRC_IPS_V1",
"src_ip_ranges": ["{{customer_nonprod_subnets}}"],
"description": "default rule"
},
{
"action": "deny",
"priority": "2147483647",
"versioned_expr": "SRC_IPS_V1",
"src_ip_ranges": ["{{all_ips}}"],
"description": "default deny rule"
}
]
}