How check if a list has any duplicate value(s)?

Hi there!

I’m trying to do an input validation, where I need to make sure values are unique in the list and should return false if not - any idea how to do that efficiently? So far. I could come up with this:

> ! can(length(var.my_list) == length(distinct(var.my_list)))
false

but doesn’t look any good at all. Couldn’t find anything in the forum either. What are my options?

-San

This example you’ve shared is returning whether it’s possible to compare the length of two collections for equality, so I expect this expression will always return false because it’s always possible to compare two numbers (and it’s negated).

However, the part you have inside the can call seems like a reasonable answer to me, reading as “the length of the list should remain the same after removing any duplicates”:

  validation {
    condition     = length(var.my_list) == length(distinct(var.my_list))
    error_message = "All elements must be unique."
  }

If this is a collection of unique items that aren’t in any particular order then a different answer could be to declare the type as being a set rather than a list. Sets automatically coalesce duplicate elements by definition, so you wouldn’t need a validation rule at all in that case:

variable "example" {
  type = set(string)
}

An advantage of this approach is that a potential user of your module can see from the type of the variable that it’s intended to be an unordered set rather than an ordered list. However, it won’t work if you need to preserve the order of elements given by the caller, because list is the only collection kind that preserves ordering.

1 Like

I didn’t realize that’s the case - I tested with a negative set and that came out false, as expected and thought should be okay. Thanks for pointing out, @apparentlymart