Interpolating output of templatefile() possible?

I have a need to construct resource names at runtime; a variable to a module has a list of user names, and I need to translate that list into a list of resource names to obtain the value of an attribute of that resource. I’m trying to use templatefile() for this, but don’t seem to be getting the result I’d expect.

github_user.tmpl

data.github_user.${user}.username

module/main.tf

resource github_team_membership "team" {
  for_each = toset(var.maintainers)
  username = templatefile("${path.root/github_user.tmpl", { user = each.value } )
}

The templatefile() function is called as I expect, and produces the proper output, however the output is treated as literal string input for the username argument, instead of being treated as a reference to a resource’s attribute as it would if I had typed it into the file directly.

Am I misunderstanding the proper behavior here? I expected the output of templatefile() to be treated identically to the same text as if it had been present instead of the function call.

Unfortunately yes, you are misunderstanding the template function - it produces text output not indirect rsources references. The typical use-case for templatefile is the user_data script for instance resources which is run at boot time.

Have you tried doing the github_user resource with a for_each loop over the maintainers list ? Outputting a map which would then be used in the membership’s for_each loop.

Thanks. Yes, I’ve tried this as well:

github_user_map.tmpl

{
%{ for user in users ~}
${user} = github_user.${user}.username
%{ endfor ~}
}

test.tf

locals {                                                                                                                                                                  
  test_map = templatefile("${path.root}/github_user_map.tmpl", {users = ["foo", "bar"]})
}

resource "github_repository_collaborator" "test" {
  for_each = local.test_map
  username = each.value
}

This results in validate reporting:

Error: Invalid for_each argument

  on test.tf line 18, in resource "github_repository_collaborator" "test":
  18:   for_each = local.test_map

The given "for_each" argument value is unsuitable: the "for_each" argument
must be a map, or set of strings, and you have provided a value of type
string.

Wrapping the string in tomap() doesn’t help, as Terraform complains it can’t convert a single string to a map (which is understandable).

At this point I think I’m just going to have to do the templating outside of Terraform in order to make progress.

Hi @kpfleming,

It looks like your goal here is to retrieve some data about a set of users in your var.maintainers and then make use of it in a team membership. That’s not a problem that templatefile will help you solve, but here’s a different way to do it:

variable "maintainers" {
  # It looks like you're only using this variable as
  # a set, so might as well just declare it as one so
  # the module caller can see that the ordering isn't
  # important.
  type = set(string)
}

data "github_user" "maintainer" {
  for_each = var.maintainers

  username = each.value
}

resource "github_team_membership" "team" {
  for_each = data.github_user.membership

  username = each.value.username
  # etc, etc
}

Notice that the for_each expression for the membership is data.github_user.membership rather than var.maintainers. That means that inside that block each.value is an object representing a data "github_user" result, and so you don’t need to do anything special to select the object by username.

This works because for any resource block where for_each is set the resulting value is a map of objects, where the keys of that map are the keys in the for_each expression. That means that your data source results will have addresses like data.github_user.maintainer["username"].


With that said, I assumed above that var.maintainers is a set of GitHub usernames, in which case retreiving the github_user objects this way would seem unnecessary: you already know the usernames and could just use them directly in github_team_membership like this:

variable "maintainers" {
  type = set(string)
}

resource "github_team_membership" "team" {
  for_each = var.maintainers

  username = each.key
  # etc, etc
}

If your goal here was to connect some other identifier used within your organization with github usernames, there’s a slight variant of the above where you could set var.maintainers to be a mapping from your internal identifiers to GitHub’s usernames. For the sake of example, I’m going to say that these “internal identifiers” are corporate email addresses under example.com:

variable "maintainers" {
  type        = map(string)
  description = "Mapping from corporate email address to GitHub username."
}

resource "github_team_membership" "team" {
  for_each = var.maintainers

  username = each.value.username
  # etc, etc
}

Notice that only the type of var.maintainers has changed above. Let’s say for example that the value of var.maintainers is something like this:

{
  "kevin.p.fleming@example.com" = "kpfleming"
  "martin.atkins@example.com"   = "apparentlymart"
}

In the above, we’d end up with resource addresses like github_team_membership.team["kevin.p.fleming@example.com"], but the username inside would be kpfleming, which thus allows the broader Terraform configuration to talk about the users in terms of their corporate email addresses while making sure that they are all mapped to the right GitHub users when interacting with GitHub’s API.

1 Like

thanks! I need to get lunch before I review this, but your last point is exactly my situation: I need to maintain configuration based on corporate identifiers, partly because GitHub identifiers are not stable and shouldn’t be used in configuration at all :slight_smile:

More to come after I experiment with this, but the most important thing you taught me here is that ‘each.value’ can be a reference to an object (data source or resource), it doesn’t have to just be a string.

(now that I’ve had some food)

This is very helpful indeed. I’d end up creating a set of ‘data.github_user’ objects in the module for each repository which has its own maintainer list, so there’s be a lot of duplicated objects since many of our employees are maintainers of more than one repository. The github_user data source is pretty expensive as well (it pulls a lot of information I don’t need, but I can change that), but this technique is worth a try.

There is another issue, unrelated to this topic though, which is that I really need to store unique (numeric) IDs for GitHub users in the Terraform configuration since GitHub usernames (logins) are not stable and when a user changes their username the next Terraform run can result in bad changes being made. I’m trying to avoid actually having to put the numeric IDs in by hand, so I’ve been experimenting with a github_user resource which can be imported (using the username) and then stores the numeric ID in the state.

I’m going to continue experimenting, this has been very useful!

The .membership trick isn’t working for me. Terraform complains that it can’t find a data resource “github_user” “membership”, so it doesn’t appear to be treating ‘membership’ as a special keyword in that context.

I found a solution that I’m happy with:

locals {
  users = {
    kflemin1 = "kpfleming"
    bbg-test = "bbg-tf-test5"
    jgrout6 = "jasongrout"
  }
}

data "github_user" "gh_user" {
  for_each = local.users

  username = each.value
}

locals {
  maintainers = ["kflemin1", "bbg-test"]
}

resource "github_team" "test" {
  name = "test"
}

resource "github_team_membership" "test-members" {
  for_each = local.maintainers

  team_id = github_team.test.id
  username = data.github_user.gh_user[each.value].username
}

So to some degree I’m still constructing the desired data source (resource) name at runtime, but it’s being done using a key into a map, which is a fully supported operation.

Thanks for making me think of alternatives! Now I can use only company-issued identifiers throughout the Terraform configuration, and translate them to GitHub identifiers where needed (and when I need to, I’ll be able to use data.github_user.gh_user[each.value].id to get the numeric user ID instead of the username.

For completeness if anyone tries this at home… I needed to be able to pass that map of data.github_user objects into modules where the real work is done. Declaring a variable for the module of type = map (no mapped type) and then passing data.github_user.gh_user worked just fine.

Oh sorry, I see that I made an editorial error while I was preparing the example to share here. It should’ve been data.github_user.maintainer to refer to the data "github_user" "membership" block, not data.github_user.membership.

1 Like

Yep, but I got there in the end :slight_smile: