Collection function introduce cannot be determined until apply

I get this error using “concat” (no error without concat function),


Error: Invalid count argument

  on .../main.tf line xx, in resource "iam_role_policy_attachment" "custom":
     count = var.create_role && length(var.custom_role_policy_arns) > 0 ? length(var.custom_role_policy_arns) : 0

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.


for the below terraform configuration, no error removing concat function, no error using


terraform apply -target='module.mod1_acc1.module.iam_policy_mod1_backup1'
terraform apply


	[Terraform live configuration]
 
module "mod1_acc1" {
  source = "../test/mod1"
  mod1 = {}
}


	[mod1 - main.tf]

module "mod2" {
  source = "../mod2"
  create_role             = true
  role_name               = "myRoleName"
  custom_role_policy_arns = compact(
      [
        module.iam_policy_mod1_backup1.arn
      ]
    )
}

module "iam_policy_mod1_backup1" {
  source = "../mod3"
  name        = "polName"
  path        = "/"
  description = "description"
  policy = data.aws_iam_policy_document.iam_policy_mod1_backup1.json
}
data "aws_iam_policy_document" "iam_policy_mod1_backup1" {
  version = "2012-10-17"
  statement {
    sid       = "sidName"
    effect    = "Allow"
    resources = ["*"]
    actions = [
      "ssm:GetParametersByPath",
    ]
  }
}


	[mod2 - main.tf]

resource "aws_iam_role" "this" {
  count = var.create_role ? 1 : 0
  name                 = var.role_name
  path                 = var.role_path
  assume_role_policy = ""
}
resource "aws_iam_role_policy_attachment" "custom" {
  count      = var.create_role && length(var.custom_role_policy_arns) > 0 ? length(var.custom_role_policy_arns) : 0
  role       = aws_iam_role.this[0].name
  policy_arn = "arn"
}


	[mod3 - main.tf]

resource "aws_iam_policy" "policy" {
  name        = var.name
  path        = var.path
  description = var.description
  policy      = var.policy
}

Where is the concat function in that code?

concat compact

In this configuration, it appears that the value for module.iam_policy_mod1_backup1.arn inside the concat argument is coming from the managed resource module.iam_policy_mod1_backup1.aws_iam_policy.policy.

This means that during planning the module.iam_policy_mod1_backup1.arn value will be unknown. Because the value is unknown, the result of the concat function will be unknown, since it cannot be determined until apply time how many non-empty values there will be in the list.

Using the list directly works, because the number of items is known, even if they were to end up as null values.

Thank you @jbardin. If functions cannot be used for values not know until apply, how can I achieve this whitout running TF many times (or using -target option) to create such resources before reference them in functions ?

Below what I like to use. Can you depict alternative code for such situations ?

module "mod2" {
  source = "../mod2"
  create_role             = true
  role_name               = "myRoleName"
  custom_role_policy_arns = concat(
    compact(
      [
        module.iam_policy_mod1_backup1.arn,
        module.iam_policy_mod1_backup2.arn,
        var.account_is_a_dr ? module.iam_policy_mod1_backup3.arn : "",
      ]
    ),
    var.iam_policy_extra_arns,
  )
}

One way some modules do is accept the length value as a parameter (rather than using the length function). That way there are no unknowns for the count, so it works.

It’s not that functions cannot be used in general, but you are using a particular function in a way where the result cannot be determined while planning. The value for count must be known, yet the arguments you are passing to concat mean that the length of the result is unknown.

In this case I would suggest restructuring the configuration so that compact is not needed, and you can use a known length to determine the correct value. From the sections of configuration you’ve shared, there doesn’t seem to be any reason this couldn’t be achieved, as even if the resources in the modules are conditionally created in some way the conditions themselves must be based on known values.

I explane more logic behind the code and ask you a question about TF.

To make code as DRY as possible I used a single module and dependency inversion to configure similar security environments for three account types involved in a backup & disaster recovery solution:

  1. backupper: one or more accounts (normally one) with an ec2 appliance assuming specific role of backed and dr accounts (with specific policies allowing backup operations)

  2. backed: one or more accounts allowing to assume its specific role to need backuppers to do backup of its resources (EC2, etc.)

  3. dr: disaster recovery accounts are as backed accounts but need specific permissions in addition to backed (e.g. dr policy)

all three types need these 2 policies:

module.iam_policy_mod1_backup1.arn
module.iam_policy_mod1_backup2.arn

only type (3.) need module.iam_policy_mod1_backup3.arn, from this i used

var.account_is_a_dr ? module.iam_policy_mod1_backup3.arn : ""

as per dependency inversion live configuration passes only “account_is_a_dr” to make module reusable for all types. The latter can be empty so I used compact to allow type (3.) conditionally.

in addition (but this is not the focus here), it can be useful to pass extra arns as permission from live configuration, so I added:

var.iam_policy_extra_arns,

and concat to allow also this extra arns (if defined in live configuration).

Certainly I can simplify and split cases 1., 2. and 3. into different modules (probably 2 will be enough) duplicating lot of code, or adding to all types special permissions now used only for dr, so compact will not be needed anymore without lot of duplication, but probably I’ll face same issue with concat and iam_policy_extra_arns too (i need to try this).

As it works if I create module.iam_policy_mod1_backup1 and module.iam_policy_mod1_backup1 with -target before apply entire configuration, what I ask you is: it’s not an option for TF to catch this issue and decide if a “function” should be evaulated based on plan or apply data so creating resources in right order (since here we do not have a dependency cycle) ?

Sorry, I’m not sure how to continue here, but I’ll try and address your specific question, which may help with any remaining confusion.

Terraform is attempting to evaluate everything as accurately as possible. The fact that when the arguments to compact are unknown, the length of the result of compact is logically unknown as well is not avoidable. If you are certain that the number of elements should be known, then there is absolutely no reason to use compact.

In order for terraform to plan the creation of resources, it must know how many of those resources it is planning, and known values must be used to determine that. It’s possible that in the future we may have some way of running progressive applies in a single execution, but until then it needs to be done manually with multiple configurations or the use of -target.

Thank you @jbardin for the explanation, got it.

Terraform is a great tool and do a great job to find dependencies and “to do all” in one apply. Probably it be of great value also a progressive applies behavior.

Sure I lack lot of knowledge of the internal terraform evaluation logic probably known in deept only with source code experience and knowledge.