External Data source cannot use simple json

Terraform
1

Hi All,

I’m working on the AWS EFS file systems. Previously our approach was very simple create File systems and access points in a single file and folder level.

I’ve tried using Data aws_efs_Access_points block and was successful only if File system already exists so I went with the External data source block with Python script. when i execute the Python script it returns the values as expected but when executing in Terraform it errors out as below.

Error: Unexpected External Program Results with data.external.efs_filesystem, on main.tf line 45, in data "external" "efs_filesystem":
│   45:   program = ["python", "./get_efs_filesystem_id.py", "service-config"]

│ The data source received unexpected results after executing the program. Program output must be a JSON encoded map of string keys and string values.
If the error is unclear, the output can be viewed by enabling Terraform's logging at TRACE level. Terraform documentation on logging:    
│ https://www.terraform.io/internals/debugging
│ Result Error: invalid character 'C' looking for beginning of value

I’ve run export TF_LOG=trace to see what’s going on I can see reulst being generated something like this.

2023-09-05T12:02:26.115+0100 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/external\"] (close)"
2023-09-05T12:02:27.377+0100 [TRACE] provider.terraform-provider-external_v2.3.1_x5.exe: Executed external program: @module=external program="C:\Users\
ss\AppData\Local\Programs\Python\Python39\python.exe ./get_efs_filesystem_id.py service-config" tf_rpc=ReadDataSource tf_provider_addr=registry.terraform.io/hashicorp/external tf_req_id=f6ac883a-164c-7013-9037-c343a99d4aaf @caller=github.com/terraform-providers/terraform-provider-external/internal/provider/data_source.go:187 output="Command-line arguments: service-config
{"filesystem_id": "fs-0ed9b40bd123456z"}" tf_data_source_type=external timestamp=2023-09-05T12:02:27.377+0100

Terraform code

data "external" "efs_filesystem" {
  for_each = local.restructured_access_points_map
  program = ["python", "./get_efs_filesystem_id.py" ]
  query = {
    filesystem_name = each.value.file_system
  }
}

Python code

import json
import sys
import subprocess
import argparse

filesystem_name = sys.argv[1]
print("Command-line arguments:", filesystem_name)

# Use the AWS CLI to query EFS for the file system ID based on the name
result = subprocess.run(
    ["aws", "efs", "describe-file-systems", "--query", f"FileSystems[?Name=='{filesystem_name}'].FileSystemId | [0]"],
    stdout=subprocess.PIPE,
    text=True
)

filesystem_id = result.stdout.strip().strip('"')

# Return the result as JSON
response = {
    "filesystem_id": filesystem_id
}
# Convert the response dictionary to JSON format
json_response = json.dumps(response)

# Print the JSON response
print(json_response, end='')

# Flush the output to ensure there's no extra whitespace
sys.stdout.flush()

Terraform version 1.0.7
can anyone assist with this please.
Thank you

2

Hi @skkc2,

The error invalid character 'C' looking for beginning of value is telling you that the process output is not valid json, and encountered the C character before it could decode a json value. This would be because your program starts by printing Command-line arguments:, which is not valid json and where the C is found at the start of the output.