Iterate over a list produced by a data source using Python `for` loop

Terraform CDK for Terraform
1

Hey all.

I have the following code:

from constructs import Construct
from cdktf import TerraformStack, TerraformOutput, Token, Fn, TerraformLocal
from cdktf_cdktf_provider_aws.provider import AwsProvider
from cdktf_cdktf_provider_aws.data_aws_organizations_organization import DataAwsOrganizationsOrganization
from cdktf_cdktf_provider_aws.data_aws_eks_clusters import DataAwsEksClusters


class DiscoveryStackConfig:
    aws_profile: str
    shared_config_files: list = None
    shared_credentials_files: list = None

    def __init__(self, aws_profile: str, shared_config_files: list = None, shared_credentials_files: list = None):
        self.aws_profile = aws_profile
        self.shared_config_files = shared_config_files
        self.shared_credentials_files = shared_credentials_files


class DiscoveryModuleStack(TerraformStack):
    def __init__(self, scope: Construct, name: str, config: DiscoveryStackConfig):
        super().__init__(scope, name)

        # Default AWS shared config and credentials files
        shared_config_files = ["~/.aws/config"] if config.shared_config_files is None else config.shared_config_files
        shared_credentials_files = [
            "~/.aws/config"] if config.shared_credentials_files is None else config.shared_credentials_files

        # Management account AWS provider definition
        AwsProvider(self, "management-account-provider", profile=config.aws_profile,
                    shared_config_files=shared_config_files, shared_credentials_files=shared_credentials_files)

        organization = DataAwsOrganizationsOrganization(self, "organization")
        accounts = Token.as_list(organization.accounts)
        TerraformOutput(self, "accounts", value=accounts)

        for idx, account in enumerate(accounts):
            account_id = Fn.lookup(Fn.element(accounts, idx), "id", "")
            TerraformOutput(self, f"account_{idx}", value=Fn.lookup(Fn.element(accounts, idx), "id", ""))
            account_provider = AwsProvider(self, f"aws.{idx}", profile=account_id, alias=f"_{idx}",
                                           shared_config_files=shared_config_files,
                                           shared_credentials_files=shared_credentials_files)

            clusters = DataAwsEksClusters(self, f"clusters_{idx}", provider=account_provider)
            TerraformOutput(self, f"clusters_output_{idx}", value=clusters)

First, I’m listing the accounts in an AWS Organization, then in each account, I’d like to list the EKS clusters.
Since each account requires a different AwsProvider definition (because provider block in Terraform doesn’t support count or for_each), I’m using Python for loop to iterate over the accounts list that is produced by the DataAwsOrganizationsOrganization data source, and create AwsProvider in each iteration.
I then use the account ID in the profile argument of AwsProvider (my AWS config file has the account IDs as the profile names).
I have 2 accounts in said accounts list, as can be seen in the TerraformOutput identified as accounts:

accounts = tolist([
 {
   "arn" = "arn:aws:organizations::222222222222:account/o-1234567890/111111111111"
   "email" = "linked1@a.com"
   "id" = "111111111111"
   "name" = "linked1"
   "status" = "ACTIVE"
 },
 {
   "arn" = "arn:aws:organizations::222222222222:account/o-1234567890/222222222222"
   "email" = "payer@a.com"
   "id" = "222222222222"
   "name" = "payer"
   "status" = "ACTIVE"
 },
])

However, when I iterate over the accounts list from DataAwsOrganizationsOrganization, only one of my 2 accounts is iterated over, as can be seen in the TerraformOutput identified as account_{idx}:

account_0 = "222222222222"

When printing the accounts Python variable (which is the one I’m iterating over), I see a list with only one element:

['#{TfToken[TOKEN.14]}']

Is there any reason that when iterating over the accounts list that is produced by DataAwsOrganizationsOrganization, using Python for loop, only the first element is extracted, as opposed to using the same list as TerraformOutput, which shows 2 elements?

Is there a way to use Python for loop to iterate over a list that is produced by a data source?

2

Hi @dahanehud :wave:

Is there a way to use Python for loop to iterate over a list that is produced by a data source?

Unfortunately, there isn’t, as the value of data sources is only available during the Terraform plan and apply phases, CDKTF can’t know the length of that list during synth which happens before that.

As an alternative, the CDKTF concept of Iterators (docs) might help you with this.

3

Thank you @ansgarm for the response.
How would you suggest I use iterators in this specific context (meaning, to create multiple AwsProvider instances)?
There’s no for_each argument in AwsProvider to accept an iterator, and the dynamic call in an iterator requires a map, which isn’t acceptable as a value in profile and alias arguments in AwsProvider.

Any idea how can I use iterator in this case, to create multiple AwsProvider instances?

4

Hi @dahanehud :wave:

My bad, I missed the part about instantiating providers from an iterator.

I’m afraid that this does not seem to be possible as it also is not possible when working with Terraform through HCL.

Are there other ways you might be able to access the list of AWS accounts you want to manage?

5

It’s basically hinging on this: https://github.com/hashicorp/terraform/issues/24476

6

Are there other ways you might be able to access the list of AWS accounts you want to manage?

I suppose I’ll move the logic of listing accounts out of CDKTF, just using Boto3 natively.
The result will be a list of the accounts which I can iterate over using Python for, and then create an AwsProvider for each account and region combination.
This will overcome the limitation (both the limitation where AwsProvider doesn’t support for_each and the limitation where a list from a data source can’t be iterated over using Python for).

I wished to keep the logic in CDKTF, but this specific logic is simple and doesn’t involve creating resources (just listing accounts), meaning no need to manage state using Terraform, so that’s fine.

Thanks for the help!