Azure Terraform: Datasource external produces weird error

manfred0191 · November 2, 2023, 6:01pm

main.tf

    terraform {
    required_version = ">= 1.5.5"

    required_providers {
        azurerm = {
        source  = "hashicorp/azurerm"
        version = ">=3.69.0"
        }
    }
    }

    provider "azurerm" {
    features {}
    skip_provider_registration = true
    subscription_id            = "xxxx-xxxxx-xxxxx-xxxxx"

    }

    provider "azurerm" {
    features {}
    alias = "sub-fits-55070-dev-01"
    skip_provider_registration = true
    subscription_id            = "xxxx-xxxxx-xxxxx-xxxxx"
    }

    module "function_app_zis" {
    # checkov:skip=CKV_TF_1:Skipped "Ensure Terraform module sources use a commit hash"
    providers = {
        azurerm = azurerm.sub-fits-55070-dev-01
    }
    source                = "./.."
    resource_group_name   = "resourcegroup238495723985"
    function_app_name     = "myfunctionzis238495723985"
    location              = "westeurope"
    use_event_hub_trigger = false
    subnet_id             = "/subscriptions/xxxx-xxxxx-xxxxx-xxxxx/resourceGroups/rg-network/providers/Microsoft.Network/virtualNetworks/vnet-001/subnets/snet-002"
    zip_file_name         = "./functionapp/httptrigger/functionapp.zip"
    }

    resource "azurerm_resource_group" "this" {
    provider = azurerm.sub-fits-55070-dev-01
    name     = "rg-fixture"
    location = "westeurope"
    }

    data "external" "function_key" {
    program    = ["bash", "${path.module}/get_function_key.sh"]
    count      = module.function_app_zis.function_app_name != "" ? 1 : 0
    depends_on = [module.function_app_zis]
    }

    resource "azurerm_monitor_action_group" "example" {
    provider            = azurerm.sub-fits-55070-dev-01
    name                = "test1mag"
    resource_group_name = azurerm_resource_group.this.name
    short_name          = "auntmary"

    email_receiver {
        name                    = "test1email"
        email_address           = "blabla@test.com"
        use_common_alert_schema = true
    }
    azure_function_receiver {
        name                     = "test1function"
        function_app_resource_id = module.function_app_zis.function_app_id
        http_trigger_url         = "https://${module.function_app_zis.trigger_url}/api/httptrigger?code=${data.external.function_key[0].result["function_key"]}"
        function_name            = "httptrigger"
        use_common_alert_schema  = true
    }
    depends_on = [data.external.function_key]
    }

    output "function_app_http_trigger_url" {
    value = "https://${module.function_app_zis.trigger_url}/api/httptrigger?code=${data.external.function_key[0].result["function_key"]}"
    }

get_function_key.sh

    #!/bin/bash

    APP_NAME="func-myfunctionzis238495723985"
    RESOURCE_GROUP="resourcegroup238495723985"
    SUBSCRIPTION="sub-fits-55070-dev-01"
    FUNCTION_NAME="httptrigger"

    # Check if the function app is deployed
    for i in {1..30}; do
        FUNCTION_KEY=$(az functionapp function keys list --subscription $SUBSCRIPTION --resource-group $RESOURCE_GROUP --name $APP_NAME --function-name $FUNCTION_NAME --query default --output tsv)
        if [ "$FUNCTION_KEY" != "" ]; then
            echo "{\"function_key\": \"$FUNCTION_KEY\"}"
            break
        fi
    done

Debug Output

Output 1st run:

    Error: Unexpected External Program Results
    │
    │ with data.external.function_key[0],
    │ on main.tf line 138, in data "external" "function_key":
    │ 138: program = ["bash", "${path.module}/get_function_key.sh"]
    │
    │ The data source received unexpected results after executing the program.
    │
    │ Program output must be a JSON encoded map of string keys and string values.
    │
    │ If the error is unclear, the output can be viewed by enabling Terraform's
    │ logging at TRACE level. Terraform documentation on logging:
    │ https://www.terraform.io/internals/debugging
    │
    │ Program: /usr/bin/bash
    │ Result Error: unexpected end of JSON input

Output 2nd run:

    Error: Unexpected External Program Results
    │
    │ with data.external.function_key[0],
    │ on main.tf line 138, in data "external" "function_key":
    │ 138: program = ["bash", "${path.module}/get_function_key.sh"]
    │
    │ The data source received unexpected results after executing the program.
    │
    │ Program output must be a JSON encoded map of string keys and string values.
    │
    │ If the error is unclear, the output can be viewed by enabling Terraform's
    │ logging at TRACE level. Terraform documentation on logging:
    │ https://www.terraform.io/internals/debugging
    │
    │ Program: /usr/bin/bash
    │ Result Error: invalid character '{' after top-level value

Output 3rd run: no errors

I am deploying a function app in Azure via Azure DevOps / terraform.

As the resource azurerm_linux_function_app has no output for the function key of the function, which is needed in the action group, because otherwise the function is not triggered via the action group, I have to use a trick to get the function key into the action group.

As you can see I try to get the function key with an az query and use this output to get it delivered in the action group.

When I am running this, in the 1st run I am getting the output shown above, which indicates to me that there is no JSON content returned.

In a second run, I added two lines to the shell script:

    az login --service-principal -u ... -p ... --tenant ...
    az account set --subscription "sub-fits-55070-dev-01"

When I am running this, I get the error message shown above in run 2. This indicates to me, that there is an output, but terraform cannot handle this output properly.

In the third run, I am changing the shell script back to what it was in run 1, which means delete the two lines again that I have added in run 2.

Now, the third run is giving me the correct output and creates the action group.

This behavior is weird, because in the third run I am running exactly the same files that I had already used in run 1. But without the az login command in between, it does not work.

What is wrong here?

Apart from that is there another way to get the function key as an output except the one I am using?

apparentlymart · November 2, 2023, 10:26pm

Hi @manfred0191,

I’m not familiar with the services you are using here, so I’m focusing only on the question of why this error might appear.

I can see at least two ways this script might fail to generate a valid JSON result:

if the Azure CLI printed an error to its stderr and exited with an unsuccessful status code. Your script doesn’t check whether the program is successful, so in that case it would be treated as producing an empty string, which would then mean your script prints nothing at all. This seems to match the first symptom.
if the CLI prints any character that would not be valid inside a JSON string then the script would try to generate JSON but would produce something invalid, which could cause the second error.

Normally I’d suggest running the script directly to confirm whether it’s behaving as expected, since then you would be able to see exactly what output it is producing. However, it sounds like your step 2 has inadvertently changed the remote system in some way that makes the first example behave differently, so you might now not be able to easily reproduce that initial error situation.

Hopefully someone else in the forum who is more familiar with these services can suggest a way to do this without running an external script at all, but if the external script is the only option then I suggest adding some error handling to the script so that it fails in a way that Terraform can understand as an error, which means exiting with a nonzero exit code (exit 1, for example), at which point I think the external data source will try to read some of the content from stderr to display as an error message.

manfred0191 · November 3, 2023, 6:47am

Hey man, thanks for your answer, which was highly appreciated.
I know that there is no error handling. I have a different script that is looking like this:

#!/bin/bash

APP_NAME="func-myfunctionzis238495723985"
RESOURCE_GROUP="resourcegroup238495723985"
SUBSCRIPTION="sub-fits-55070-dev-01"
FUNCTION_NAME="httptrigger"

az login --service-principal -u b88*****  -p TC_**** --tenant 121***

az account set --subscription "sub-fits-55070-dev-01"

# Check if the function app is deployed
for i in {1..30}; do
    FUNCTION_KEY=$(az functionapp function keys list --subscription $SUBSCRIPTION --resource-group $RESOURCE_GROUP --name $APP_NAME --function-name $FUNCTION_NAME --query default --output tsv | tr -d '[:space:]')
    if [ "$FUNCTION_KEY" != "" ]; then
        echo "{\"function_key\": \"$FUNCTION_KEY\"}"
        exit 0  # Exit with success status code
    fi
    sleep 1  # Add a short delay between retries
done

# If no key is found after 30 tries, exit with an error message
echo "{\"error\": \"Unable to retrieve function key\"}"
exit 1

But I was confused by Azure’s behavior and the script that I published here was the one that was at least one time running correctly, so I changed my git back to this script.

Actually, the thing is, and now listen carefully:

All of these scripts, the first one or this one above, are working completely correct when I run them out of a local linux shell. I am getting this:

{"function_key": "9brtKRKYOUG7al0AqSx-I4iWnmzP-r3gKVKIa_XYxuFuAzFu77iJxA=="}

So the key questions in fact are:

Why are all these scripts running perfectly from my local machine and none of them is working correctly when run from the pipeline in Azure DevOps?
What is actually changed in the system like you are saying correctly, that the third run is doing what is should do, but that I need the two failed runs in advance? This is the key question.
Is there a way to get the function key without this shell script? Maybe with an ARM template deployment with terraform or with whatever? This “whatever” would help me indeed!