Iterate through a nested map in templatefile function

Terraform
1

Hello HashiCommunity,

I am wondering if anybody would be aware of a way to iterate through a map in the templatefile function directly (nested in a for_each).

Let me explain by giving the case which I am currently working on.

I aim to create an AWS SNS module that generates some SNS topics and attaches an access policy to them.

Here’s what I got so far:

MAIN.TF

resource "aws_sns_topic" "sns_topics" {
  for_each = var.sns_topics   /* A map that contains the names of the topics to be created */
  name     = each.value.name
}

resource "aws_sns_topic_policy" "sns_topic_policy" {
  for_each = var.sns_policies /* A map that contains the same key names as  the ones in sns_topics */

  arn    = aws_sns_topic.sns_topics[each.key].arn
  policy = templatefile(each.value.filepath,{
      aws_account_id = var.aws_account_id
      topic_name     = each.value.topic_name
      policy_id      = each.value.policy_id
      sid            = each.value.variables["access_policy"].sid
      effect         = each.value.variables["access_policy"].effect
      action         = each.value.variables["access_policy"].action
      resource       = each.value.variables["access_policy"].resource
     [...]
    }
  )
}

Since a single access policy can contain multiple sets of sid, effect, action, resource, …,
I understand that targeting a specific key in the map each.value.variables is not the best way to achieve what I am looking for.

I am looking for a way (maybe a wildcard character or a Terraform trick that I would not be aware of) to iterate through each.value.variables for_each keys that it contains.

Any tips would be greatly appreciated.

Let me know if you need any other information regarding the case.

Thank you !

2

Hi @stillnocake,

I think I understand why the example you shared doesn’t meet your needs (you want to do something with the potentially-any elements of each.value.variables), but I’m not sure about what you want to achieve with those elements.

It might help if you can show an example of what sort of value you’d like that templatefile call to produce, and the corresponding example of a var.sns_policies element value, and then hopefully from that I can give some suggestions on how to get that desired result in a way that generalizes over other values following that structure.

3

Hi @apparentlymart , I apologize for the late reply.
We will consider a case where var.sns_policies is defined like:

terraform.tfvars

sns_policies = {
  "topic1" = {
    topic_name = "topic1"
    policy_id = "__default_policy_ID"
    filepath  = ".\\templates\\policies\\sns\\topic.json.tmpl" 
    variables = {
      "access_sid1" = {
        sid             = "sid1"
        effect          = "Allow"
        principal_key   = "AWS"
        principal_value = "*"
        action          = [
          "SNS:GetTopicAttributes",
          "SNS:SetTopicAttributes",
          "SNS:AddPermission",
          "SNS:RemovePermission",
          "SNS:DeleteTopic",
          "SNS:Subscribe",
          "SNS:ListSubscriptionsByTopic",
          "SNS:Publish",
          "SNS:Receive"
        ]
        resource       = ["arn:aws:sns:us-east-1:735542588967:topic1"]
        operator       = "StringEquals"
        operator_key   = "AWS:SourceOwner"
        operator_value = "735542588967"
      }
    }
  },
  "topic2" = {
    topic_name = "topic2"
    policy_id = "__default_policy_ID2"
    filepath  = ".\\templates\\policies\\sns\\topic.json.tmpl" 
    variables = {
      "access_sid1" = {
        sid             = "sid1"
        effect          = "Allow"
        principal_key   = "AWS"
        principal_value = "*"
        action          = [
          "SNS:GetTopicAttributes",
          "SNS:SetTopicAttributes",
          "SNS:AddPermission",
          "SNS:RemovePermission",
          "SNS:DeleteTopic",
          "SNS:Subscribe",
          "SNS:ListSubscriptionsByTopic",
          "SNS:Publish",
          "SNS:Receive"
        ]
        resource       = ["arn:aws:sns:us-east-1:735542588967:topic2"]
        operator       = "StringEquals"
        operator_key   = "AWS:SourceOwner"
        operator_value = "735542588967"
      },
      "access_sid2" = {
            sid  = "sid2"
            effect ="Deny"
            principal_key   = ""
            principal_value = ""
            action = [
                "sns:DeleteEndpoint",
                "sns:CreatePlatformEndpoint",
                "sns:GetEndpointAttributes",
                "sns:ListPlatformApplications"
            ]
            resource       = "*"
            operator       = ""
            operator_key   = ""
            operator_value = ""
        }
    }
  }
}

We can see that there can be multiple sids in a single statement and I would like to be able to support that feature. As I understand, my issue comes from the fact that since I already used for_each in the main, I am not able to iterate through the variables map nested in var.sns_topics. This forces me to target a specific key (access_policy) in the variables map making me asking for a way to do it iteratively with this post.
I have modified this part (access_policy) to access_sid# since then to illustrate my use case clearer to you.

Additionally, since other keys of the template are sometimes unused, I would just leave them as empty strings and deal with the exception in the main.tf using the “?” operator hoping that the template would not create an empty configuration block in the final policy.
Are you aware of another way I could use to do such a thing?
Could dynamic blocks or an equivalent be used inside the templatefile function?

Here is the template I will be working with:

topic.json.tmpl

${jsonencode({
  "Version": "2008-10-17",
  "Id": policy_id,
  "Statement": [
    {
      "Sid": sid,
      "Effect": effect,
      "Principal": {
        "${principal_key}": principal_value
      },
      "Action": [for act in action : "${act}"],
      "Resource": [for res in resource : "${res}"],
      "Condition": {
        "${operator}": {
         "${operator_key}": operator_value
        }
      }
    }
  ]
})}

Everything has already been tested and works for a single sid case.

Again, I am asking if you are aware that the function templatefile() would support any kind of for loop. If not, I will have to think of another way to make things work.

Thank you very much for your help and have a nice day !

4

Thanks for that extra information! So what I understand now is that you want to have one aws_sns_topic_policy instance per entry in var.sns_policies, which you already achieved, but then for each of those policies to include potentially many statements as part of the JSON value.

I think the best way to proceed here would be to just pass the whole variables data structure into the template as a single template argument, and then use the different parts of that structure within the jsonencode argument inside your template.

The resource block would then look something like this:

resource "aws_sns_topic_policy" "sns_topic_policy" {
  for_each = var.sns_policies

  arn    = aws_sns_topic.sns_topics[each.key].arn
  policy = templatefile(each.value.filepath, {
    aws_account_id = var.aws_account_id
    topic_name     = each.value.topic_name
    policy_id      = each.value.policy_id
    statements     = each.value.variables
  })
}

The template you shared already had some extra complexity in it that I think wasn’t really needed, so before I show the final template I want to show a simplified version that I’m starting with, just so you can see more clearly how the final template differs from this interim simplification step:

${jsonencode({
  "Version": "2008-10-17",
  "Id": policy_id,
  "Statement": [
    {
      "Sid": sid,
      "Effect": effect,
      "Principal": {
        (principal_key): principal_value
      },
      "Action": action,
      "Resource": resource,
      "Condition": {
        (operator): {
         (operator_key): operator_value
        }
      }
    }
  ]
})}

What I did here was remove the unneeded interpolation sequences and for expressions, which weren’t making any significant impact on the result.

Here’s a version updated for the resource block I showed above, where the statements come in all together as a single template variable statements:

${jsonencode({
  "Version": "2008-10-17",
  "Id": policy_id,
  "Statement": [
    for stmt in statements : {
      "Sid": stmt.sid,
      "Effect": stmt.effect,
      "Principal": {
        (stmt.principal_key): stmt.principal.value
      },
      "Action": stmt.action,
      "Resource": stmt.resource,
      "Condition": {
        (stmt.operator): {
          (stmt.operator_key): stmt.operator_value
        },
      },
    }
  ]
})}

What I did here was to turn the literal, single-element Statement list into a for expression over the elements of statements. The expression for each element is the same as I showed above except that it now refers to e.g. stmt.sid instead of just sid because the for expression puts the currently-selected statement in the stmt temporary symbol.

Your original didn’t really make any use of the keys of this variables map, so I didn’t use them here either, but I did just want to note that if you remove sid from the individual objects and just use the map key as the sid instead then you can potentially simplify the input data structure slightly, though it will require a slightly different for expression in the template in order to access the map keys:

  # ...
  "Statement": [
    for sid, stmt in statements : {
      "Sid": sid,
      # ...

This two-symbol form of a for expression places the element key in sid and the element value in stmt, so you can use sid directly to get the statement ID while you continue to use stmt to access all of the other attributes as in my example above. That’s optional though; if you have a reason to keep the map keys and the sids separate then by all means just keep my original answer!

5

Hello again @apparentlymart ,
First and foremost, let me thank you for your amazing answer and the level of optimization you brought in to my rather clunky solution.

I had no idea that a complex map of objects like variables could have been considered as a map by Terraform’s templatefile() as it does not contain only strings (I was under the feeling it wouldn’t be because of my previous experiences with Terraform playing with maps).

Also, I was not aware of the existence of the parenthesis interpolation syntax you used to reference to the value of the json keys. It maybe was due to a lack of json knowledge on my part, I really don’t know.

Regarding the use of the keys in the variables map, you are absolutely spot-on and I hugely thank you for that. This will save some lines to an already hefty configuration file ^^.

To top everything up, since I have now only one template for every use cases, I think I that I will be able to remove the filepath for each topics and use a single attribute for it. This is simply mindblowing !

Thanks a million for all the time and effort you’ve put into your answers !

P.S.: I have spotted a small typo in the code you offered me. stmt.principal.value should’ve been read stmt.principal_value .
Everything works now ! :smiley: