I recently provisioned an AKS cluster with private mode enabled using terraform aks module. Considering my use case, I found the kubenet network plugin most appropriate.
I’m working, full private with a Hub-Spoke network topology across multi subscriptions.
What I mean by that is that I got 1 Hub vnet (with vpn gateway, private dns zone and appligation gateway) in a subscription and 1 Spoke vnet (with private aks cluster). Vnets are peered. I want to glue everything so that private dns entries, appgw config and aks work together.
Now everything is provisioned fine. Using External dns and Agic, appgw and dns zone updates work fine but some details are confusing and I can’t find information about these specific topics out there.
I don’t understand why I can’t manually set an Private endpoint IP address. The only parameter is an enable that triggers the creation of a private endpoint over the kube apiserver and the private ip address is picked from the default_node subnet. I find this inconvenient, what’s preventing me from having the private endpoint over a different subnet ?
Why Application gateway is pointing to k8s pod IPs ? Why not k8s Services IP ? This kind of resource on the k8s side would be more appropriate.
Specifically using Kubenet, any idea why a routing table is required ? Supposing you got 2 vnets peered, no address space overlap. Traffic leaving appgw towards aks should be routed (by default) to the right destination ?
This last question initially brought me here as the required kubenet route table forces traffic to an aks node IP as network virtual appliance.
Now I technically don’t know from what reference/ datasource obtain this vmss instance private IP to provide it to the UDR definition.
EDIT: Apparently not supported since IP can change between terraform applies: 2211
I am probably miss reading/understanding some of the docs I’ve read but any opinion on these subjects would be much appreciated.
Hope explanation is clear enough, I’ll edit if required.
Many thanks !