Help building hierarchy of resources (e.g. GitLab group/subgroups) from a map variable

Terraform
1

TL;DR: I’d like to build a hierarchy of resources from details in a locals.map_variable that are stitched together via id’s that are generated at “apply-time” and I think I’m asking for the impossible.

I’d like to assemble a set of GitLab groups/subgroups using the GitLab provider’s gitlab_group resource.

Subgroups are linked to their parent via the parent’s id, parent_id.

It’s straightforward to bang these out with an explicit set of resources, simply referring to the parent group and grabbing its ID in the subgroup.

I’m using a module to create the groups with appropriate defaults, it’s a riff and extension of ideas from Lotte-Sara Laan’s Module Parameter Defaults with the Terraform Object Type post. I can just add a separate invocation of my module for each resource, but I end up needing to call terraform init every time I add a new one, which seems unintuitive.

I’d rather just define the set of groups in a map variable and invoke the module once using a for_each loop.

The module is invoked something like this:

module "groups" {
  source         = "./modules/defaulted_gitlab_group"
  for_each       = local.our_groups
  name           = each.value["name"]
  path           = each.value["path"]
  description    = each.value["description"]
  parent_id      = each.value["parent_id"]
  group_settings = each.value["group_settings"]
}

Ideally our_groups would be something like this:

locals {
  # various things elided...
  our_groups = {
    "foo" : {
      name           = "Foo"
      path           = "foo"
      description    = "Foo"
      group_settings = local.group_defaults
      members = {
        "george-hartzell" = "owner",
      }
    }
    "foo-bar" : {
      name           = "Foo Bar"
      path           = "bar"
      description    = "Foo Bar"
      group_settings = local.group_defaults
      parent_id      = module.groups["foo"].gitlab_group.id
    }
  }
}

BUT, I end up with an error message like this (edited from a real example to match the sanitized input above):

╷
│ Error: Unsupported attribute
│
│   on foo_groups.tf line 35, in locals:
│   35:       parent_id      = module.groups["foo"].gitlab_group.id
│     ├────────────────
│     │ module.groups["foo"] is a object, known only after apply
│
│ This object does not have an attribute named "gitlab_group".
╵

I’ve also tried a second approach, where the local map contains the name of the parent group and I look it up in the module call/definition:

module "groups" {
  source         = "./modules/defaulted_gitlab_group"
  for_each       = local.our_groups
  name           = each.value["name"]
  path           = each.value["path"]
  description    = each.value["description"]
  group_settings = each.value["group_settings"]
  parent_id      = module.groups[each.value["parent_group_name"]].group.id
}

Somewhat predictably, this leads to a cycle:

╷
│ Error: Cycle: module.groups (close), module.groups.var.parent_id (expand), module.groups.gitlab_group.group, module.groups.output.group (expand)
│

The only other resource I found that supported a hierarchy is a Google folder but I was unable to find any examples of people generating them from a map like I’m trying to do.

I’d love suggestions for getting out of the corner I’ve painted myself into.

Thanks!

2

Reading more, I think that there might be a solution based on the approach described in Flattening nested structures for for_each where I arrange the groups into a hierarchy in the map variable.

I’ll keep digging, any thoughts or encouragement (or waving off) would be appreciated.

3

Well, that was a false alarm. I got my hopes up when it spoke of “nested data structure” and thought it would give a way around hierarchy. I now think not.

I’ve also found the Terraform issue titled Allow dynamically-recursive child module references with count and for_each #27248, which I think describes what I need to solve my problem (as a Don’t Get Your Hopes Up enhancement).

I’ll try to follow up with @apparentlymart 's request for additional use cases there:

Likewise, if anyone else finds this enhancement request and has another use-case to share, please add a comment describing it and showing what you’ve tried so far and what didn’t work.

4

Hi @hartzell,

I don’t think you’ll be able to achieve exactly what you’re looking for with Terraform, because it calls for a dynamic dependency graph between instances of the same resource, whereas Terraform’s dependency graph is between whole resources and the individual instances only appear once Terraform has visited the resource and evaluated the for_each expression.

I think the closest you’ll be able to get here is to have the groups themselves each defined with individual resources, and then to collect them all up into a map so that you can refer to them dynamically for other declarations later:

locals {
  groups = {
    foo     = gitlab_group.foo.id
    foo-bar = gitlab_group.foo-bar.id
  }
}

Using this you could at least then systematically declare the group memberships in terms of your data structure, even though the groups themselves are individually defined:

locals ={
  group_members = flatten([
    for name, group in local.our_groups : [
      for username, role in group.members : {
        username   = username
        role       = role
        group_name = name
      }
    ]
  ])
}

resource "gitlab_group_membership" "test" {
  for_each = {
    for gm in local.group_members : "${gm.username}:${gm.group_name}" => gm
  }

  group_id     = local.groups[gm.group_name].id
  user_id      = gitlab_user.all[gm.username].id
  access_level = gm.role
}
5

@apparentlymart – thanks for the followup!

I can see how that would work, though it would end up forgoing the GitLab group/subgroup structure. We could replicate a lot of the “inheritance” benefits by just inheriting settings and etc w/in the Terraform config, but we’d lose e.g. rolling issues up to higher group levels.

I’ll think about it a bit, the other path forward is to just hardcode a hierarchy for the GitLab groups/subgroups/projects that I really want to “manage as code” and then leave the rest of the world to sleep in the bed it makes.

As usual, I’ve learned a lot painting myself into this particular corner. Thanks for the great explanation!

6

I was also trying to find a solution for this issue, I ended up generating Terraform code recursively from a hierarchically defined Gitlab groups config file.
I hope to find a better solution, some day.