Dynamic policy generation - Error: "policy" contains an invalid JSON: invalid character '"' after array element

Hi all,

I hope someone can point me in the right direction.

I am trying to build a KMS key policy. In this policy I need to allow several GuardDuty detector IDs from different regions. So my desired policy should look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGuardDutyKey",
      "Effect": "Allow",
      "Principal": {
          "Service": "guardduty.amazonaws.com"
      },
      "Action": "kms:GenerateDataKey",
      "Resource": "arn:${aws_partition}:kms:${kms_key_region}:${source_account}:key/${kms_key_id}",
      "Condition": {
        "StringEquals": {
            "aws:SourceAccount": "${source_account}",
            "aws:SourceArn": [
                "arn:${aws_partition}:guardduty:${guardduty_detector_region}:${source_account}:detector/${guardduty_detector_id_1}",
                "arn:${aws_partition}:guardduty:${guardduty_detector_region}:${source_account}:detector/${guardduty_detector_id_2}",
                "arn:${aws_partition}:guardduty:${guardduty_detector_region}:${source_account}:detector/${guardduty_detector_id_N}"
            ]
        }
      }
    }
  ]
}

I need to build the aws:SourceArn condition key, which is a list, dynamically depending on the number of GuardDuty connector IDs that I am passing.

So far this is the configuration I have:

In main.tf:

locals {
  guardduty_detector_ids = [
    {"eu-west-1"      = data.aws_guardduty_detector.ireland_guardduty_detector.id},  # returns a detector ID - works fine
    {"us-east-1"      = data.aws_guardduty_detector.virginia_guardduty_detector.id},
    {"eu-central-1"   = data.aws_guardduty_detector.frankfurt_guardduty_detector.id},
    {"eu-west-2"      = data.aws_guardduty_detector.london_guardduty_detector.id},
    {"eu-west-3"      = data.aws_guardduty_detector.paris_guardduty_detector.id},
    {"eu-north-1"     = data.aws_guardduty_detector.stockholm_guardduty_detector.id},
    {"us-east-2"      = data.aws_guardduty_detector.ohio_guardduty_detector.id},
    {"us-west-1"      = data.aws_guardduty_detector.north_california_guardduty_detector.id},
    {"us-west-2"      = data.aws_guardduty_detector.oregon_guardduty_detector.id},
    {"ca-central-1"   = data.aws_guardduty_detector.central_guardduty_detector.id},
    {"ap-south-1"     = data.aws_guardduty_detector.mumbai_guardduty_detector.id},
    {"ap-northeast-3" = data.aws_guardduty_detector.osaka_guardduty_detector.id},
    {"ap-northeast-2" = data.aws_guardduty_detector.seoul_guardduty_detector.id},
    {"ap-southeast-1" = data.aws_guardduty_detector.singapore_guardduty_detector.id},
    {"ap-southeast-2" = data.aws_guardduty_detector.sydney_guardduty_detector.id},
    {"ap-northeast-1" = data.aws_guardduty_detector.tokyo_guardduty_detector.id},
    {"sa-east-1"      = data.aws_guardduty_detector.sao_paulo_guardduty_detector.id}
  ]
}

module "aws_guardduty_findings_kms_key" {
  source                  = "./kms-key"
  description              = "This is used to encrypt GuardDuty exported findings"
  policy = templatefile("policy/aws_guardduty_findings_kms_key_policy.json", {
    source_account = var.account_id
    kms_key_region = var.aws_region
    kms_key_id = "*"
    guardduty_detector_ids = local.guardduty_detector_ids
    aws_partition = "aws"
  })
}

In policy/aws_guardduty_findings_kms_key_policy.json file I have:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGuardDutyKey",
      "Effect": "Allow",
      "Principal": {
          "Service": "guardduty.amazonaws.com"
      },
      "Action": "kms:GenerateDataKey",
      "Resource": "arn:${aws_partition}:kms:${kms_key_region}:${source_account}:key/${kms_key_id}",
      "Condition": {
        "StringEquals": {
            "aws:SourceAccount": "${source_account}",
            "aws:SourceArn": [
              %{ for guardduty_detector_id_info in guardduty_detector_ids }
              %{ for guardduty_detector_region, guardduty_detector_id in guardduty_detector_id_info }
                "arn:${aws_partition}:guardduty:${guardduty_detector_region}:${source_account}:detector/${guardduty_detector_id}"
              %{ endfor }
              %{ endfor }
            ]
        }
      }
    }
  ]
}

As you can see, I am using the templatefile() function to build a template policy where I can dynamically construct the list value of the aws:SourceArn condition key, so I can get my desired policy as shown above.

But when I apply the above configuration, I get an error:

Error: "policy" contains an invalid JSON: invalid character '"' after array element
│
│   with module.aws_guardduty_findings_kms_key.aws_kms_key.kms_key,
│   on ./kms-key/main.tf line 5, in resource "aws_kms_key" "kms_key":
│    5:   policy                   = var.policy

Can someone please point out what am I doing wrong here? I don’t understand why this is failing and what is should I be doing differently. It seems that it doesn’t like something about the double quotes that the ARN value is wrapped in.

If I put the entire statement inside the two for loops, then it works fine. But that will increase a lot the size of the policy as it will create the statement for each item in the local guardduty_detector_ids variable, which is not ideal.

I would appreciate very much some guidance.

Thank you very much in advance!

At a glance, it looks your for loop which constructs strings does not include any commas, so it builds invalid JSON like:

[
"arn:foo:guardduty:..."
"arn:foo:guardduty:..."
"arn:foo:guardduty:..."
]

This is a common problem with building JSON payloads using template files, which is why in general we recommend using the jsonencode function, which can encode an arbitrary tree of data into a well-formatted JSON string.

Note that some AWS objects depend on key order, which the jsonencode function will not allow you to specify. You may find the aws_iam_policy_document data source suitable for your needs.

If neither of those options works for you, try using the join function with your constructed template strings to ensure they’re separated by commas.

Hope this helps!

Hi @alisdair

Thank you for taking the time to answer.

I tried also adding a comma at the end of the ARN, but that also fails with the below error:

"policy" contains an invalid JSON: invalid character ']' looking for beginning of value

I have a usecase where I need to use a template file that’s why I’m using the templatefile() function.

When you say that it is better to use the jsonencode() function, how would I go about using that in my template? Could you give an example?

As for the join() function, I don’t think I can use that as I am looping over a list of maps, and then in a nested loop, I’m going over the key:item of each map. So I don’t have a list of string to use with the join() function. At least I don’t see how I could use it.

Thank you again!

If you’ve just added a comma to the end of every line then again it will be invalid JSON as the final entry must not have a comma. Getting the formatting correct can be quite a pain, which is why using jsonencode() is much easier. You just need to pass in the list that you’ve created (via whatever for expressions, etc. needed) and it returns a string you can then put within your template.

Ideally if this is just a complete JSON file (rather than some JSON within a file with other content) you wouldn’t use templatefile() at all and just create the right object within Terraform and then encode. Even if you do have some JSON embedded within something else in a template I’d just have the template substitute a single string for the whole section, again created via jsonencode()

Hi @stuart-c

Thanks for the reply which pointed me actually in the right direction. I’ve managed to fix it and now have the right configuration and works as expected.

For anyone else wanting to do the same or something similar, I’m posting the final configuration below.

The locals and the module call now look like this (I moved the for loops inside the locals block instead of having them inside the template):

guardduty_detector_ids = [
    {"${var.aws_region}"              = data.aws_guardduty_detector.ireland_guardduty_detector.id},
    {"${var.virginia_region}"         = data.aws_guardduty_detector.virginia_guardduty_detector.id},
    {"${var.frankfurt_region}"        = data.aws_guardduty_detector.frankfurt_guardduty_detector.id},
    {"${var.london_region}"           = data.aws_guardduty_detector.london_guardduty_detector.id},
    {"${var.paris_region}"            = data.aws_guardduty_detector.paris_guardduty_detector.id},
    {"${var.stockholm_region}"        = data.aws_guardduty_detector.stockholm_guardduty_detector.id},
    {"${var.ohio_region}"             = data.aws_guardduty_detector.ohio_guardduty_detector.id},
    {"${var.north_california_region}" = data.aws_guardduty_detector.north_california_guardduty_detector.id},
    {"${var.oregon_region}"           = data.aws_guardduty_detector.oregon_guardduty_detector.id},
    {"${var.central_region}"          = data.aws_guardduty_detector.central_guardduty_detector.id},
    {"${var.mumbai_region}"           = data.aws_guardduty_detector.mumbai_guardduty_detector.id},
    {"${var.osaka_region}"            = data.aws_guardduty_detector.osaka_guardduty_detector.id},
    {"${var.seoul_region}"            = data.aws_guardduty_detector.seoul_guardduty_detector.id},
    {"${var.singapore_region}"        = data.aws_guardduty_detector.singapore_guardduty_detector.id},
    {"${var.sydney_region}"           = data.aws_guardduty_detector.sydney_guardduty_detector.id},
    {"${var.tokyo_region}"            = data.aws_guardduty_detector.tokyo_guardduty_detector.id},
    {"${var.sao_paulo_region}"        = data.aws_guardduty_detector.sao_paulo_guardduty_detector.id}
  ]

  guardduty_detector_arns = flatten([
    for guardduty_detector_id_info in local.guardduty_detector_ids : [
      for guardduty_detector_region, guardduty_detector_id in guardduty_detector_id_info : [
        "arn:${var.aws_partition}:guardduty:${guardduty_detector_region}:${var.account_id}:detector/${guardduty_detector_id}"
      ]
    ]
  ])

module "aws_guardduty_findings_kms_key" {
  source                  = "./kms-key"
  description              = "This is used to encrypt GuardDuty exported findings"
  policy = templatefile("policy/aws_guardduty_findings_kms_key_policy.json", {
    source_account = var.account_id
    kms_key_region = var.aws_region
    kms_key_id = "*"
    guardduty_detector_arns = local.guardduty_detector_arns
    aws_partition = "aws"
  })
}

The JSON file with the policy template now looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGuardDutyKey",
      "Effect": "Allow",
      "Principal": {
          "Service": "guardduty.amazonaws.com"
      },
      "Action": "kms:GenerateDataKey",
      "Resource": "arn:${aws_partition}:kms:${kms_key_region}:${source_account}:key/${kms_key_id}",
      "Condition": {
        "StringEquals": {
            "aws:SourceAccount": "${source_account}",
            "aws:SourceArn": ${jsonencode("${guardduty_detector_arns}")}
        }
      }
    }
  ]
}

Hope it helps someone.

Thank you both for the help!

If you are just making a policy document is there any reason you aren’t using Terraform Registry ?

The templatefile documentation page has an example of using jsonencode as the entire result of a template.