Deduplicate Map in For Loop

Terraform
1

Consider the following example:

resource aws_route53_record acm_validation {
  for_each = {
    # use all the info as a key (which is unused) to ensure deduplication of records
    for dvo in aws_acm_certificate.default.domain_validation_options : "${dvo.resource_record_name}/${dvo.resource_record_type}/${dvo.resource_record_value}" => {
      name = dvo.resource_record_name
      record = dvo.resource_record_value
      type = dvo.resource_record_type
    }
  }

  name = each.value.name
  records = [each.value.record]
  ttl = var.dns_ttl
  type = each.value.type
  zone_id = var.zone.id
}

Note that I’m trying to form a map where the key is:

"${dvo.resource_record_name}/${dvo.resource_record_type}/${dvo.resource_record_value}"

The reason I’m using such a long key is to de-duplicate records in the final map created. ACM certificate validations often contain duplicate records and I’d like to de-duplicate to not create/update the exact same record twice.

Terraform reports an error:

Error: Duplicate object key
…
Two different items produced the key “…”
If duplicates are expected, use the ellipsis (…) after the value expression to enable grouping by key

In a simpler example in terraform console:

{ for k in ["a", "a"] : k => 1000 }

The error in full:

│ Error: Duplicate object key
│
│   on <console-input> line 1:
│   (source code not available)
│
│ Two different items produced the key "a" in this 'for' expression. If duplicates are expected, use the ellipsis (...) after
│ the value expression to enable grouping by key.

If I do group by key, like so:

{ for k in ["a", "a"] : k => true ... }

I get a result that I don’t want:

{
  "a" = [
    true,
    true,
  ]
}

I simply want a merged output with duplicate keys superseded.

How can I deduplicate this in my above expression so no duplicate records are present in the map?

EDIT

I tried wrapping it in distinct and no dice:

{ for dvo in distinct(aws_acm_certificate.default.domain_validation_options) : ... }

Still throws an error.

2

Maybe look at flatten() and / or merge()? Could look at Unable to map the value when i have nested map variable - #2 by wyardley, which may not match your use case exactly, but might provide some ideas about how to debug it. This will also show you roughly how I’ve usually used the spread operator.

If you want further help, maybe provide an example with a list of maps or objects with some actual plausible values for the structure you’re trying to create?

3

Yes, I think to your comment what I could do would be to convert each entry in the map to a list and then distinct on that list somehow as a filter.

4

If you’re building a map with a compound key, I think that will already somewhat dedupe things (you don’t have to actually use that compound key, and can then flatten back to a list).

High level, I’ve found that approach onlined in that post (with a self-contained tf file) to be really helpful to me in iterating quickly on HCL data structures, since in Terraform. Also, just figuring out what final data structure is going to work best for your use case sometimes takes a bit of thinking / adjustment.

5

I think this is more side effect than limitation, because “aws_acm_certificate.default” can be same “domain” when created the certificate.

Example : domain: example.com alternate_domain: example.com

Can you share with us the state object of this certificate?

6

I think probably the smallest change compared to what you already tried would be to use the ... symbol to group by key as you tried, and then use the last element of each of the groups when you assign the arguments:

resource aws_route53_record acm_validation {
  for_each = {
    # use all the info as a key (which is unused) to ensure deduplication of records
    for dvo in aws_acm_certificate.default.domain_validation_options : "${dvo.resource_record_name}/${dvo.resource_record_type}/${dvo.resource_record_value}" => {
      name = dvo.resource_record_name
      record = dvo.resource_record_value
      type = dvo.resource_record_type
    }...
  }

  name    = element(each.value, -1).name
  records = [element(each.value, -1).record]
  ttl     = var.dns_ttl
  type    = element(each.value, -1).type
  zone_id = var.zone.id
}

If you’d rather not duplicate that element call for each argument separately then you can get a similar effect by pre-flattening in a local value:

locals {
  all_domain_validation_options = {
    # use all the info as a key (which is unused) to ensure deduplication of records
    for dvo in aws_acm_certificate.default.domain_validation_options : "${dvo.resource_record_name}/${dvo.resource_record_type}/${dvo.resource_record_value}" => {
      name = dvo.resource_record_name
      record = dvo.resource_record_value
      type = dvo.resource_record_type
    }...
  }
  flat_domain_validation_options = {
    for k, v in local.all_domain_validation_options : k => element(v, -1)
  }
}

resource aws_route53_record acm_validation {
  for_each = local.flat_domain_validation_options

  name    = each.value.name
  records = [each.value.record]
  ttl     = var.dns_ttl
  type    = each.value.type
  zone_id = var.zone.id
}

I’m afraid I don’t have a good one-liner for this situation.

7

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.