Hybrid connection network does not send data when done via Terraform and works when done manually

Terraform Providers Azure
1

We have a challenge with creating hybrid connections via Infra structure as code and associating these with our app service.

We deploy our infrastructure with Terraform (AzureRM) and experience that everything is created correctly and also seems to be wired up ok.

But what we have observed is that our application service is able to connect the the on-premise endpoint but does not receive any data. It manifests itself by our code getting a timeout when trying to talk to our endpoint.

We know that the app service can see the endpoint as we can tcpping it on its from Kudu

PS C:\home> tcpping b2b.te7.datahub.dk:443
tcpping b2b.te7.datahub.dk:443
Connected to b2b.te7.datahub.dk:443, time taken: 73ms

The hybrid connection in the app service looks ok and also says it is connected:
image

What we have found out is that if we disconnect the hybrid connection in the app service. Wait a minute and then add the hybrid connection again manually, the app service is able to communicate with our on-premise resource just fine.
So it seems that something in the setup of the hybrid connection via IAC is failing.

We have tried to compare the arm template with the setup deployed via IAC and how it looks like when it is done manually. We have not found any significant differences between the two:
image

When the HC is deployed via IAC it is done with a service connection and service principal that has the Contributor role.

We have tried quite a few approaches to setting up the HC via IAC. For example, we have tried using the RootManageSharedAccessKey and also trying to setup dedicated shared access keys on the HC itself with the same outcome.

Our main theory right now is that there is something in the timing of the creation of the HC and how soon it is associated with the app service that breaks something, but that’s just a shot in the dark. Right now we are limited to associating the HC manually but this is not ideal as we want our production environment only to be accessible to humans with the ‘Reader’ role.

To provide insight here is the Terraforms that we use to commission the app and HC:

image
image853×483 14.3 KB

1 Like
2

Hi @kristianschneider ,
have you tried to create each resource step-by-step?
This might also reveal a dependency or timing issue.

Does your deployment logs show if the HC is created early compared to the relay itself ?

3

Hi @tbugfinder

Thanks for your reply.
How would you do it step-by-step? Without dependencies between the components in terraform?

I’ll try and look at the order in the logfiles

4

Well, either you try the apply -target switch.

  -target=resource       Resource to target. Operation will be limited to this
                         resource and its dependencies. This flag can be used
                         multiple times.

Or you start within the .tf file using these steps:
a) define resource A
b) terraform apply
c) add resource B
d) terraform apply
e) add resource C
f) terraform apply
…

5

Hi @kristianschneider, Have you found a solution to this issue? I have just tried to setup the same thing and encounter exactly the same issue as you were describing here.

6

Nope. Had to do it manually,:expressionless:

7

Oh… FYI - I worked around the issue by executing azure cli commands to remove the hybrid connection from the webapp and then adding it back again via “local-exec” provisioner:

resource “azurerm_web_app_hybrid_connection” “web_app_hybrid_connection” {
… azurerm_web_app_hybrid_connection properties here…

provisioner “local-exec” {
# Call Azure CLI Script here
command = “az webapp hybrid-connection remove --hybrid-connect HybridConnectionName --resource-group {var.RESOURCE_GROUP} --namespace AzureRelayNamespace --name {azurerm_windows_web_app.app_service.name} && az webapp hybrid-connection add --hybrid-connect HybridConnectionName --resource-group {var.RESOURCE_GROUP} --namespaceAzureRelayNamespace --name {azurerm_windows_web_app.app_service.name}”
}
}