Dynamic, try, and yamldecode

Terraform
1

I tried few tests on how to use dynamic, try, and yamldecode on google_artifact_registry_repository to configure remote_repository_config.

# repos.yaml
- name: new-team
  description: Container image repo for New team
  repo_owners:
    - group1@test.com
  labels:
    owner: new-team
  region: us-east1
  repo_format: DOCKER
- name: pypi
  description: Repository of software for the Python programming language
  repo_owners:
    - group1@test.com
    - group2@test.com
    - group3@test.com
  labels:
    owner: group1
  region: us-east1
  repo_format: PYPI
  cleanup_policy_dry_run: true
  mode: REMOTE_REPOSITORY
  repo:
  - description: "PyPi Remote Repository"
    repository: PYPI


# vars.tf
locals {
  repositories = flatten([
    for each_repo in yamldecode(file("repos.yaml")) : {
      name                   = each_repo.name
      description            = each_repo.description
      region                 = each_repo.region
      repo_format            = each_repo.repo_format
      labels                 = each_repo.labels
      repo_owners            = each_repo.repo_owners
      cleanup_policy_dry_run = tobool(each_repo.cleanup_policy_dry_run)
      mode                   = lookup(each_repo, "mode", "STANDARD_REPOSITORY")
      repo                   = try(each_repo.repo, [])
    }
  ])
  xrepos = distinct(flatten([
    for each_repo in local.repositories : [
      for repo in each_repo.repo : {
        repo = try(repo, [])
      }
    ]
  ]))
}

# repositories.yaml
resource "google_artifact_registry_repository" "repository" {
  for_each               = { for repo in local.repositories : "${repo.name}" => repo }
  location               = each.value.region
  repository_id          = each.value.name
  description            = each.value.description
  format                 = each.value.repo_format
  labels                 = each.value.labels
  cleanup_policy_dry_run = each.value.cleanup_policy_dry_run
  mode                   = each.value.mode
  dynamic "remote_repository_config" {
    for_each = try(local.xrepos, {})
    content {
      description = remote_repository_config.value.repo["description"]
      python_repository {
        public_repository = remote_repository_config.value.repo["repository"]
      }
    }
  }
...
...

From the above, in repos.yaml, the new-team was created a while back, and at that time, we did not consider REMOTE_REPOSITORY, so, I refactored repositories.yaml to add mode, repo and xrepos.

The problem with that code is the existing new-team will be deleted and recreated. Sample of terraform plan

  # google_artifact_registry_repository.repository["pypi"] will be created
  + resource "google_artifact_registry_repository" "repository" {
      + cleanup_policy_dry_run = true
      + create_time            = (known after apply)
      + description            = "Repository of software for the Python programming language"
      + effective_labels       = {
          + "owner" = "cloud-engineering"
        }
      + format                 = "PYPI"
      + id                     = (known after apply)
      + labels                 = {
          + "owner" = "group1"
        }
      + location               = "us-east1"
      + mode                   = "REMOTE_REPOSITORY"
      + name                   = (known after apply)
      + project                = "GOOGLE-PROJECT-1"
      + repository_id          = "pypi"
      + terraform_labels       = {
          + "owner" = "group1"
        }
      + update_time            = (known after apply)

      + cleanup_policies {
          + action = "DELETE"
          + id     = "delete-untagged"

          + condition {
              + package_name_prefixes = []
              + tag_prefixes          = []
              + tag_state             = "UNTAGGED"
              + version_name_prefixes = []
            }
        }
      + cleanup_policies {
          + action = "KEEP"
          + id     = "keep-3-tagged"

          + most_recent_versions {
              + keep_count            = 3
              + package_name_prefixes = []
            }
        }

      + remote_repository_config {
          + description = "PyPi Remote Repository"

          + python_repository {
              + public_repository = "PYPI"
            }
        }
    }

  # google_artifact_registry_repository.repository["new-team"] must be replaced
-/+ resource "google_artifact_registry_repository" "repository" {
      ~ create_time            = "2023-06-30T18:08:14.610188Z" -> (known after apply)
      ~ id                     = "projects/GOOGLE-PROJECT-1/locations/us-central1/repositories/new-team" -> (known after apply)
      ~ name                   = "new-team" -> (known after apply)
      ~ update_time            = "2024-01-09T15:36:24.318084Z" -> (known after apply)
        # (10 unchanged attributes hidden)

      + remote_repository_config { # forces replacement
          + description = "PyPi Remote Repository" # forces replacement

          + python_repository { # forces replacement
              + public_repository = "PYPI" # forces replacement
            }
        }

        # (2 unchanged blocks hidden)
    }

I hope I explained the problem clear enough.

Any helps is highly appreciated.

2

Hi @laurentiuspurba,

I’m afraid I’ve not yet studied all of this close enough to make a full proposal for what to do here, but I did want to note something that I think reflects a misunderstanding:

You’ve written try(local.xrepos, {}), but this is a “useless use of try” because local.xrepos can never fail as long as there’s a local value called xrepos, and it will always fail (during static validation, rather than at runtime) if there is not such a local value. try is only for errors that arise dynamically based on runtime values.

I’m not sure I follow exactly what you intended that to mean, but if your goal is to decide dynamically how many remote_repository_config blocks to generate then you’ll need to write the expression for the xrepos local value to produce a different number of elements depending on the situation. If you make that evaluate to an empty collection then that would cause there to be no remote_repository_config blocks generated, which I assume is your goal based on the plan output you shared.

3

Hi @apparentlymart

Thanks for the response.

My intention is to create remote_repository_config if the following are defined in repos.yaml:

  • mode: is REMOTE_REPOSITORY and
  • repo: is configured

I really appreciate your pointers on this. I did check your previous responses from other posts and couldn’t find that is similar to this or my lack of understanding, I guess.

Thanks!
-Laurentius

4

Thanks for the extra context!

An important part of this requirement that I didn’t understand at first is that each google_artifact_registry_repository should have either zero or one remote_repository_config blocks based on some attributes in each.value. That means that the dynamic block for_each will need to decide its result based only on each.value data, rather than referring back to the entire original collection of repository settings.

If you had the luxury of changing the YAML structure then I’d suggest changing it so that a remote repository is specified by including a nested YAML map in a single item called “remote” (or similar) that would be unset/null for non-remote repositories, because Terraform has some language features to help with translating that shape onto a dynamic block condition concisely, and because that structure better matches the shape of the underlying resource schema.

However, it sounds like you are retrofitting this in an existing system and so there are limits to how much you can change the existing structure. It’s possible to describe the rule you specified as an expression for for_each in your dynamic block, but it’s a little clunky:

  for_each = (
    each.value.mode == "REMOTE_REPOSITORY" && each.value.repo != null ?
    each.value.repo :
    null
  )[*]

This is using the conditional operator to choose between returning the “repo” value or null, and then using the “splat” operator [*] to turn that into a zero or one element tuple to match what for_each is expecting.

For this to work you should also change your fallback value for when repo isn’t set in the YAML to be null rather than [] – since that’s the usual way to represent absence for a single value rather than a collection of values – and make the repo values in the YAML docs also provide only a single value rather than a YAML list. Then repo will be a single value that is null when not set, which is what my example above is expecting.

Inside the content block you can use remote_repository_config.value to refer to that repository’s repo attribute value, if any. (If there isn’t one then the content block need not be evaluated at all, so you don’t need to worry about that situation when defining the block content.)

I hope that helps!

5

Thanks @apparentlymart

I don’t mind modifying the repos.yaml. I came to the conclusion to have this format

mode: REMOTE_REPOSITORY
repo:
- description: "PyPi Remote Repository"
  repository: PYPI

was because the mode: can be:

  • STANDARD_REPOSITORY
  • REMOTE_REPOSITORY
  • VIRTUAL_REPOSITORY

and then, those values above, except STANDARD_REPOSITORY drives the repository: value.

I appreciate if you could suggest something that applicable to the above.

And I will try your suggested solution using existing format.

Thank you,
-Laurentius

6

Thanks for that additional context. I’m not really familiar with this particular resource type, so I was making some guesses about what might be appropriate, but I think with what you’ve shared now I would suggest the following…

Since mode is already being passed to the provider and it’s laid out in the resource type schema as a separate argument from the remote_repository_config block, I think the most direct representation in YAML would be to treat repo as optional (null if not set) and use whether it’s set or not as the entire rule for whether that nested block would be included.

The YAML then would change slightly so that repo is just a plain YAML map, rather than a list with one map inside it:

mode: REMOTE_REPOSITORY
repo:
  description: "PyPi Remote Repository"
  repository: PYPI

Your logic for interpreting the YAML data into a well-typed Terraform data structure is mostly fine as-is, but I’d tweak the handling of repo to use null as the fallback when repo isn’t present:

  repositories = flatten([
    for each_repo in yamldecode(file("repos.yaml")) : {
      name                   = each_repo.name
      description            = each_repo.description
      region                 = each_repo.region
      repo_format            = each_repo.repo_format
      labels                 = each_repo.labels
      repo_owners            = each_repo.repo_owners
      cleanup_policy_dry_run = tobool(each_repo.cleanup_policy_dry_run)
      mode                   = lookup(each_repo, "mode", "STANDARD_REPOSITORY")
      repo                   = try(each_repo.repo, null)
    }
  ])

Then in your resource block you can use the null-ness of repo to decide between zero or one remote_repository_config blocks like this:

  dynamic "remote_repository_config" {
    for_each = each.value.repo[*]
    content {
      description = remote_repository_config.value.description
      python_repository {
        public_repository = remote_repository_config.value.repository
      }
    }
  }

The [*] operator there is a concise conversion from a single value that might be null into a tuple of either zero or one elements, to suit what for_each expects.

7

Hi @apparentlymart

I will give a shot your suggestion. I really appreciate this.

Thank you,
Laurentius