Import resources to state file during provisioning

Terraform
1

Hi,

Is it possible to import resources that are implicitly created, while provisioning ?

My use case :
terraform apply of azurerm_kubernetes_cluster with private_endpoint_enabled option, creates implicit resources such as dns_A_record. These resources are not created by the terraform state so its impossible to reference it and modify prior to import.
Right now, I have to manually do :
terraform import dns_A_record dns_A_record_id
and re-plan/apply to be able to update this dns record.

Tried to use local-exec to automate that import but it doesn’t seems to work this way.

Is there any way of automating / making it less ugly ?

Thanks!

2

Hi @adlnc!

What you’re basically saying is (if I understand you correctly): I want to manage the managed DNS record in a managed DNS zone. You can make it easier by managing the DNS zone yourselves.

You can Bring Your Own DNS Zone by specifying private_dns_zone_id, which brings you closer to your goal: manage the DNS record instead of let them manage it for you.

After having this Zone, managing the DNS record may work when you create it before the cluster creates it (don’t know if that works, you’d need to test it)… I expect it would work, as most operations in Azure are create/update without explicitly checking existence.

Handling and configuring the permissions for the BYO DNS Zone is documented here.

It might trigger some interesting behaviour when you update your AKS Cluster, but that is inevitable. Reconciliation will always mess with changed settings :frowning: My question regarding this setup would be what you’d like to accomplish, as I wouldn’t recommend going to much off the usual pathways.

The docs also contain an example how to configure the DNS Zone:

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "West Europe"
}

resource "azurerm_private_dns_zone" "example" {
  name                = "privatelink.eastus2.azmk8s.io"
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_user_assigned_identity" "example" {
  name                = "aks-example-identity"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
}

resource "azurerm_role_assignment" "example" {
  scope                = azurerm_private_dns_zone.example.id
  role_definition_name = "Private DNS Zone Contributor"
  principal_id         = azurerm_user_assigned_identity.example.principal_id
}

resource "azurerm_kubernetes_cluster" "example" {
  name                    = "aksexamplewithprivatednszone1"
  location                = azurerm_resource_group.example.location
  resource_group_name     = azurerm_resource_group.example.name
  dns_prefix              = "aksexamplednsprefix1"
  private_cluster_enabled = true
  private_dns_zone_id     = azurerm_private_dns_zone.example.id

  ... rest of configuration omitted for brevity

  depends_on = [
    azurerm_role_assignment.example,
  ]
}
1 Like
3

Well, thanks for your help !

Yes you absolutely got my issue. So I tried to reproduce based on your idea.
It seems ok to create the private zone and set private_dns_zone_id within azurerm_kubernetes_cluster.
Now when I try to create the dns record and then provision the aks cluster I get an error:

azurerm_kubernetes_cluster.aks: Creating…
containerservice.ManagedClustersClient#CreateOrUpdate:
Failure sending request: StatusCode=0 – Original Error: Code=“BadRequest”
Message="DNS record for FQDN subdomain ‘aks-private’ already exists on private dns zone

So it doesn’t seems possible create this dns_A_record resource before aks cluster I believe.

As a workaround I tried setting up another entry with different prefix but the issue now would be related to the SSL certificate not issued for that specific subdomain.

I’ll try to sum up the situation as best as I can.
What I’m currently trying to achieve is to setup a private aks cluster within a hub-spoke topology where I have vpn gateway, application gateway, private dns zone and hub vnet on 1 subscription and then on a different subscription, the spoke vnet and aks cluster working with aad-pod-identity, agic and external dns.

The main drawback is that I have limited number of IP available to expose resources from azure through vpn. I can’t use aks kubenet plugin because I work across multiple subscription and maintaining route tables doesn’t seems possible across subscriptions.

So I came up with the idea of having 1 subnet for aks, from which pods will get their ips, and another subnet to expose resources through vpn.
Pods get let’s say “private ips” and accessed through the application gateway. Vnets are peered so connectivity between appgw and pods is ok.

Now since the Aks cluster is given a specific subnet, the private endpoint associated to the kube apiserver is automatically within this subnet. So if I want to interact with the kube api-server privately I need a private endpoint linked to the kube api-server Private Link.
And then comes the best part, dns resolution and SSL.
As I mentioned when you set a dns_prefix parameter. This prefix is used to generate the certificate, if I want to interact with that api-server, the only option that I see is to update the dns alias with the IP of the new private endpoint.

At the current time everything seems to work. The only “manual and hacky thing” is this managed DNS record.

That’s quite a long explanation but you asked :stuck_out_tongue:
Hopefully that makes sense.
I’m open for suggestions, better ideas …

4

My networking might be even worse than yours, so don’t laugh if my solutions don’t work :smiley:

Maybe a stupid suggestion, but can’t we use multiple Private DNS Zones with the same name (containing both the same DNS record, pointing to a different private IP/Private Endpoint), and multiple Private Endpoints with the same Link? The second Private Endpoint with second DNS Zone can be created separate from the existing one as they are linked to separate VNets, although these VNets are peered.

5

Wow, that looks like a great idea.
So I found the time to test today.
2 Private dns zones with the same name.

Still I don’t really understand how it works because I created 2 private dns zone with same name (global resource, so tenant wide if I’m correct). 2 records with same prefix.
Why is it choosing one entry over the other ?
Related to the fact that de dns zone is peered to the same vnet as the vpn gateway ?
Or just because I created the entry after the one created by the aks resource ?

Didn’t have time to test yet. Anyway Thanks a lot for your help on this. I think you found a decent solution.

1 Like
6

The reason this works is because a Private DNS Zone needs to be bound to a VNet. If it is not bound to a VNet, it doesn’t work or affect your resources. Everything in your Hub is separate from the spoke, except that it is connecting to the right private link, and thus not influenced by the 2nd DNS Zone.

Your first zone is also affecting your VPN Gateway as it is landing in the right VNet. Not sure how the DNS is forwarded to your on prem network but if that works as is then I’m not complaining. That would need to be solved anyway, also with your own solution.

Great to have found a decent solution together, thanks for sharing your details!