Using External Data Source and Python imported modules

Terraform
1

Hey there,first time using the forum !

Trying to create an external data resource that runs a Python script which returns all EC2 instances id’s for a specific cluster name.

For example :
input “cluster_name”:“production”
output:“i-213sadas,i-123asdaf…”

ERROR IM HAVING :

│   File "./list_instances.py", line 1, in <module>
│     import boto3
│ ImportError: No module named boto3
│
│
│   with data.external.external_resource,
│   on main.tf line 3, in data "external" "external_resource":
│    3: data "external" "external_resource" {

Does external data resource can run imported modules …?

CODE :
main.tf

terraform {}

data "external" "external_resource" {
  program = ["python", "${path.module}/list_instances.py"]

  query = {
    cluster_name = "production"
  }
}

output "output" {
  value = data.external.external_resource.result
}

list_instances.py

import boto3
import sys
import json

client = boto3.client('ecs')

def list_containers(cluster_name): 
    response = client.list_container_instances(
        cluster=cluster_name
    )
    return response['containerInstanceArns']

def describe_container():
    query=sys.stdin.readlines()
    cluster_name="empty for now"
    response = client.describe_container_instances(
    cluster=cluster_name,
        containerInstances=list_containers(cluster_name)
    )
    instances=list(map(lambda x:x['ec2InstanceId'],response['containerInstances']))
    sys.stdout.write(json.dumps(result))


if __name__ == '__main__':
    describe_container()
2

Hi @Eyal159!

From the external provider’s perspective here, python is just an arbitrary program to run and what that program does is entirely a matter of how it’s implemented.

With that said, I don’t know if any technical reason why Python wouldn’t be able to find imported libraries when running from the external provider, as long as you’ve installed that library in one of the search paths that python is configured to use.

If you are using a mechanism like virtualenv/venv to create an isolated environment with its own set of libraries then I believe the usual way to make that work is to run the python executable that is included inside the virtual environment, in its bin directory, which therefore effectively adds the extra directories to the library search path. However, if that isn’t sufficient then you may have better results asking about this in a forum for Python rather than for Terraform, since the participants there are more likely to know the details of how venv works and how Python searches for libraries in general.

3

Hey @apparentlymart !

Thanks you for the comment.

I understand your suggestion,and it works as expected(when providing the full path)

Is there a way to see the given JSON from terraform to the program ?