Looping over nested objects from a yaml file

Terraform
1

Hi all,

Having issues walking through a yaml file and looping inside a loop. I’ve set up the yaml file to have a list of folders I want to create in gcp, the projects inside that folder I want to create and the api the projects need activated. It looks basically like this:

folders:
  folder_a:
    display_name: "Folder A"
    parent: 
    projects:
      project_a:
        api:
          - api_1
          - api_2
      project_b:
        api:
          - api_1

Looping over the folder creation is trivial (but maybe inelegant the way I’m doing it), but the projects and api are giving me headaches

resource "google_folder" "folder-structure" {
  for_each = yamldecode(file("./org_structure.yml"))["folders"]
  display_name = each.value.display_name
  parent = "organizations/${var.org-id}${each.value.parent}"
}

resource "google_project" "pmd-projects" {
  depends_on = [
    google_folder.folder-structure,
  ]
  loop over folders:
    loop over projects in folders:
      name = project.key
      project_id = project.key
      folder_id = key from first loop
}

resource "google_project_service" "project-api-service" {
  depends_on = [
    google_project.pmd-projects,
  ]

  loop over folders:
    loop over projects:
      loop over api
        project = "google_project.${project.key}.project_id"
        service = "${api}.googleapis.com"
}

Looked at flatten, but I don't think that will help here (?). Not sure if a dynamic block is the way to go either. Any help would be greatly appreciated.
2

Hi @pEdler,

This does feel like a flatten-shaped problem to me: you need to flatten a nested structure into a few different other shapes that prioritize the nested objects as the top-level elements.

Here’s an example of the deepest one for your google_project_service resource:

locals {
  folders = yamldecode(file("./org_structure.yml"))["folders"]
  folder_project_apis = flatten([
    for fk, folder in local.folders : [
      for pk, project in folder.projects : [
        for api_name in project.api : {
          folder_key  = fk
          project_key = pk
          api_name    = api_name
        }
      ]
    ]
  ])
}

resource "google_project_service" "project-api-service" {
  for_each = {
    for fpa in local.folder_project_apis : "${fpa.folder_key}/${fpa.project_key}/${fpa.api_name}" => fpa
  }

  project = google_project.pmd-projects["${each.value.folder_key}/${each.value.project_key}"].project_id
  service = "${each.value.api_name}.googleapis.com"
}

In the above I’ve constructed a slash-separated compound key for each google_project_service.project-api-service instance, like folder_a/project_a/api_1. I also assumed that there’d be a similar two-part key for the google_project.pmd-projects instances, like folder_a/project_a, but I didn’t include that directly here.

Hopefully you can use this example to write a similar process for google_project.pmd-projects, this time only flattening two levels of data structure rather than all three. If you run into troubles with that, let me know what you tried and what happened and I’ll try to help!

3

Hi @apparentlymart , I have following yml file. The purpose is to read each of the value and append some word with each of the value. Like following:-

For s3 bucket, s3-internal-data, it will be s3-internal-data-<acc_no>-
instance_profile_contains: inst-ec2-rol will be inst-ec2-rol-<acc_no>-
full_access: S3-DATA-MANAGER will be aws-<acc_no>-S3-DATA-MANAGER-

Finally I want to perform account lookup using the values above using data and if not exist, will create them otherwise will change existing attribute.

Please suggest.


schema-bucket-access:
  - s3_bucket_prefix:  s3-internal-data
    instance_profile_contains:
      - inst-ec2-rol
    iam_role_suffix:
      full_access:
        - S3-INTERNAL-DATA-ADMIN
        - S3-DATA-MANAGER
      read_only:
        - ENGINEER  
  - s3_bucket_prefix:  s3-support-data
    instance_profile_contains:
      - inst2-ec2-rol
    iam_role_suffix:
      full_access:
        - S3-SUPPORT-DATA-ADMIN
        - S3-DATA-MANAGER
      read_only:
        - ENGINEER
      conditional:
        directory_operational:
          - S3-SUPPORT-DATA-OPERATIONAL
        directory_crm:
          - S3-SUPPORT-DATA-CRM
1 Like