I have been trying to provision a Kubernetes cluster with a firewall and ingress rules using a pipeline in Azure DevOps to automatically deploy the cluster from a Github repository. It executes perfectly until it gets to the creation of the route table, which generates the following message:
Error: [0m[0m[1mupdating Route Table Association for Subnet “AzureFirewallSubnet” (Virtual Network “crec-dev-cluster-vnet” / Resource Group “crec-dev-clusteer-rg”): network.SubnetsClient#CreateOrUpdate: Failure sending request: StatusCode=400 – Original Error: Code=“AzureFirewallSubnetRouteTableMustHaveDefaultRouteToInternet” Message=“Route Table crec-dev-fwrt on firewall subnet AzureFirewallSubnet must have 0.0.0.0/0 route with next hop Internet.”
This causes the route table to be created, but the subnet association to the route table is not created, causing an additional error to be generated by the Kubernetes deployment, but this error is simply stating that the subnet needs a route table, which I am unable to associate with the subnet. Below is my Terraform manifest for creating my network components, If needed I can also include the Terraform manifest for creating the kubernetes cluster as well. From my readings the 0.0.0.0/0 route to internet is the default route, and you should be able to override it with your own 0.0.0.0/0 route but for some reason I am unable to get this to happen.
resource "azurerm_virtual_network" "crec-dev-cluster-vnet" {
address_space = ["10.40.0.0/14"]
location = var.region
name = var.vnet-name
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
}
resource "azurerm_subnet" "AzureFirewallSubnet" {
address_prefixes = ["10.41.0.0/16"]
name = var.subnet-name
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
virtual_network_name = azurerm_virtual_network.crec-dev-cluster-vnet.name
}
resource "azurerm_firewall_policy" "crec-dev-fwpol" {
location = var.region
name = "crec-dev-fwpol"
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
dns {
proxy_enabled = true
}
}
resource "azurerm_firewall_policy_rule_collection_group" "crec-dev-fwrcg" {
firewall_policy_id = azurerm_firewall_policy.crec-dev-fwpol.id
name = "crec-dev-fwrcg"
priority = 100
application_rule_collection {
action = "Allow"
name = "fqdn"
priority = 101
rule {
name = "httpfqdn"
protocols {
port = 80
type = "Http"
}
protocols {
port = 443
type = "Https"
}
source_addresses = ["*"]
destination_fqdn_tags = ["AzureKubernetesService"]
}
}
network_rule_collection {
action = "Allow"
name = "aksNetworkRules"
priority = 100
rule {
destination_ports = [1194]
name = "apiudp"
protocols = ["UDP"]
source_addresses = ["*"]
destination_addresses = ["AzureCloud.${var.region}"]
}
rule {
destination_ports = [9000]
name = "apitcp"
protocols = ["TCP"]
source_addresses = ["*"]
destination_addresses = ["AzureCloud.${var.region}"]
}
rule {
destination_ports = [123]
name = "time"
protocols = ["UDP"]
source_addresses = ["*"]
destination_fqdns = ["ntp.ubuntu.com"]
}
}
}
resource "azurerm_route_table" "crec-dev-fwrt" {
location = var.region
name = "crec-dev-fwrt"
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
}
resource "azurerm_route" "crec-dev-cluster-fwrn" {
address_prefix = "0.0.0.0/0"
name = "crec-dev-cluster-fwrn"
next_hop_type = "VirtualAppliance"
next_hop_in_ip_address = azurerm_firewall.crec-dev-cluster-firewall.ip_configuration[0].private_ip_address
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
route_table_name = azurerm_route_table.crec-dev-fwrt.name
}
resource "azurerm_route" "crec-dev-cluster-fwinternet" {
address_prefix = "${azurerm_public_ip.crec-dev-cluster-fwpublicip.ip_address}/32"
name = "crec-dev-cluster-fwinternet"
next_hop_type = "Internet"
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
route_table_name = azurerm_route_table.crec-dev-fwrt.name
}
resource "azurerm_subnet_route_table_association" "crec-dev-rta" {
route_table_id = azurerm_route_table.crec-dev-fwrt.id
subnet_id = azurerm_subnet.AzureFirewallSubnet.id
}
resource "azurerm_public_ip" "crec-dev-cluster-fwpublicip" {
allocation_method = "Static"
location = var.region
name = var.fw-public-ip
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
sku = "Standard"
}
resource "azurerm_firewall" "crec-dev-cluster-firewall" {
location = var.region
name = var.fw-name
resource_group_name = azurerm_resource_group.crec-dev-clusteer-rg.name
sku_name = "AZFW_VNet"
sku_tier = "Standard"
firewall_policy_id = azurerm_firewall_policy.crec-dev-fwpol.id
ip_configuration {
name = "crec-dev-cluster-fwconfig"
subnet_id = azurerm_subnet.AzureFirewallSubnet.id
public_ip_address_id = azurerm_public_ip.crec-dev-cluster-fwpublicip.id
}
}
I have tried defining the route both inside and outside of the route table and both produced the same errors.