Loop through 2 lists, remove matches, and group result into chunks to iterate over

Hello,

I am trying to perform some logic in Terraform, to leverage data sources, rather than have to wait for tf apply to finish, which creates an output, then have to parse the outputs in another scripting language

The goal:
(Preferably using ‘for_each’ not ‘count’)

  • Define a data source which returns a large list of permissions (over 5k)

  • Create a tf variable that houses a list of permissions to remove from the list created by the data source

  • Loop through both lists, compare each item in each list, and if the items match, remove the item from the permissions list created by the data source

  • Break the results list it into chunks of 2999 items

  • Loop over each chunk, creating a tf resource (role) per chunk, which contains all item with in that specific chunk loop

My attempt:

data "google_iam_role" "owner-role" {
  name = "roles/owner"
}

data "google_iam_testable_permissions" "non-supported-perms" {
  full_resource_name   = "//cloudresourcemanager.googleapis.com/projects/${var.prj_id}"
  custom_support_level = "NOT_SUPPORTED" # "SUPPORTED"
  stages               = ["GA", "BETA"]
}

locals {
  ownerRolePermissions = data.google_iam_role.owner-role.included_permissions
  nonSupportedPerms    = data.google_iam_testable_permissions.non-supported-perms.permissions
  customOwnerPermsList = compact(flatten([for o in local.ownerRolePermissions : [for p in local.nonSupportedPerms : [o == p ? "" : o]]])) # flatten into one list, then remove all blank entries via 'compact', to leave us with the list of permissions that are allowed in custom roles by GCP
  permissionGroup      = ceil(length(local.customOwnerPermsList) / 2999) # determine how many groups of 2999 to make
}

# Create Custom Owner Role
resource "google_project_iam_custom_role" "custom-owner" {
  #foreach     = chunklist(local.customOwnerPermsList, 2999)
  count       = local.permissionGroup
  role_id     = "GCP_Custom_Owner_${count.index}" # Must match Google Regex for role id
  title       = "GCP-Custom-Owner-${count.index}"
  description = "Custom Role ${count.index} with Owner Mirrored Permissions"
  permissions = element(chunklist(local.customOwnerPermsList, 2999), count.index) // the idea was to keep in sync with the index of groups of 2999 from 'local.permissionGroup', and grab the contents of each group in that loop
}

My attempts do not seem to be removing items from the list after comparing, and the chunk list does not seem to actually be working when resources are planned for creation

Any help is greatly appreciated, to replicate what I could typically do in scripting language

Anyone able to shed some light on how to properly achieve this in Terraform?