Template_file dynamic vars

I’m wondering if it’s possible to be able to build a vars dynamically in the template_file. I have not seen anything on it and everything from for, for_each has not worked for me using a map(string). any ideas on how to convert the below to take the user_data_vars map(string) and loop through it to take the place of vars? i’m trying to modularize this so some variables will be common but others will be environment specific according to the database that is being setup. so i want to just pass through a map(string) and set the variables to be rendered in the template.

variable "user_data_vars" {
  description = "Custom tags to set on the template file"
  type = map(string)
  default = {}
}

 user_data_vars = {
    elastic_cluster_name          = "kyle_cluster_name"
    elastic_version               = "7.0.12-1"
    elastic_dc_name               = "aw2"
  }

data "template_file" "dba_user_data" {
  count    = var.db_server_count
  template = file("${path.module}/../../../dev/services/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl")

  vars = {
    datacenter           = var.short_region
    environment          = var.environment
    environment_lower    = var.environment_lower
    salt_master          = var.salt_master
    hostname             = "${var.short_env}-${var.short_region}-${var.db_cluster_name}-${count.index + 1}.${var.domain}"
    db_cluster_name      = var.db_cluster_name
    elastic_cluster_name = var.elastic_cluster_name
    elastic_version      = var.elastic_version
    elastic_dc_name      = var.elastic_dc_name
  }

}

Hi @kstephenson,

I think what you are asking here is how to merge the elements in var.user_data_vars with the fixed set you’ve assigned to vars in your example, so that the template can use both.

The most direct answer to your question is that you can use the merge function to produce a mapping that has the elements from two other maps, like this:

  vars = merge(
    var.user_data_vars,
    {
      datacenter = var.short_region
      # etc, etc
    },
  )

Note that you can decide whether the predefined elements will take priority over the custom ones or vice-versa by changing the order of arguments to merge. Later arguments take priority over earlier ones in the event that multiple maps have elements with the same key.


With that said, I have some additional notes here that might be useful depending on what your underlying “real world” problem is here, as opposed to the specific solution you’ve decided to use within Terraform.

The first thing I want to note is that the template_file data source has been deprecated since Terraform v0.12 and continues to exist only for backward compatibility with modules written for Terraform v0.11 and earlier. Terraform now has the templatefile function built in to the Terraform language, and so you can use it to render templates from external files without depending on an external provider and in a way that integrates fully with the rest of the Terraform language, because it’s not interpreting the template in a separate plugin program.

Here’s an equivalent to what you showed in your example, modified as I suggested above, using the templatefile function instead:

locals {
  dba_user_data = templatefile(
    "${path.module}/../../../dev/services/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl",
    merge(
      var.user_data_vars,
      {
        datacenter = var.short_region
        # etc, etc
      },
    )
  )
}

This uses a local value to assign the template result to a symbol you can use elsewhere in the module, so you can replace references like data.template_file.dba_user_data.rendered with local.dba_user_data instead, with the same effect. If you only use this template result in one place in your module, you might instead choose to just inline that templatefile function call directly in that argument and not declare a separate symbol at all, but that’s a subjective style tradeoff:

resource "aws_instance" "example" {
  # ...
  user_data = templatefile(
    # (...as above...)
  )
  # ...
}

The other thing I wanted to note is that in your example you have it rendering a template that has a hard-coded path that seems to traverse outside of the module. That’s a pretty atypical pattern because it breaks the expected encapsulation of a module by making it depend on the directory structure around it. Since it seems like your goal here is to let the caller of the module provide a custom template to go with their custom template variables, I would suggest making the path to the templates also be a variable, either by passing in a path to an exact file or by passing in a prefix that you would then append your expected subdirectory structure to:

variable "template_path_prefix" {
  type = string
}

locals {
  dba_user_data = templatefile(
    "${var.template_path_prefix}/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl",
    merge(
      # (...as above...)
    )
  )
}
variable "dba_user_data_template_file" {
  type = string
}

locals {
  dba_user_data = templatefile(
    var.dba_user_data_template_file,
    merge(
      # (...as above...)
    )
  )
}

Which of the above to use will probably depend on whether you have other templates under the given prefix that this module also uses, but either way the benefit of this is that the calling module can use it’s own path.module to select a path within its own area of influence, and thus avoid the surprising encapsulation break:

module "example" {
  # ...

  template_path_prefix = "${path.module}/services"
}

What you wrote will work fine as long as the directory structures line up, but I’m suggesting this because if you have other folks working on this configuration in future who already have some Terraform experience I expect they would find it surprising for a module to internally assume a particular directory structure above its own directory, rather than being able to see the path prefix or exact filename pass explicitly through the module input variables.

thank you for the update. the merge worked well. now i have a few questions about the new templatefile. I have the below that you gave me. it seems to be missing the count parameter for the hostname variable. how can this be passed inside of this function? how do i call this local function from the aws_instance resource?

locals {
  dba_user_data = templatefile(
    "${path.module}/../../../dev/services/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl",
    merge(
      var.user_data_vars,
      {
        datacenter            = var.short_region
        environment           = var.environment
        environment_lower     = var.environment_lower
        salt_master           = var.salt_master
        hostname              = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}.${var.domain}"
        db_name               = var.db_name
        db_version            = var.db_version
        db_dc_name            = var.db_dc_name
      },
    )
  )
}

resource "aws_instance" "db_instance" {
  count     = var.db_server_count
  subnet_id = element(sort(data.aws_subnet_ids.APP_subnet.ids), count.index)
  key_name               = var.aws_key_name
  ami                    = data.aws_ami.db_ami.id
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.db_sec_grp.id]
  
  lifecycle {
    ignore_changes = [ami, instance_type, user_data_base64]
  }

  /*user_data_base64 = base64encode(
    element(
      data.template_file.dba_user_data.*.rendered,
      count.index,
    ),
  )*/

  user_data_base64 = templatefile(dba_user_data,count.index)

  dynamic "ebs_block_device" {
    for_each = var.ebs_block_device
    content {
      device_name = ebs_block_device.value.device_name
      volume_size = lookup(ebs_block_device.value, "volume_size", null)
      volume_type = lookup(ebs_block_device.value, "volume_type", null)
      iops = lookup(ebs_block_device.value, "iops", null)
      throughput = lookup(ebs_block_device.value, "throughput", null)
    }
  }

  tags = {
    Name        = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}"
    CNAME       = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}.${var.domain}."
    Application = var.app_name
    Role        = var.db_type
    Project     = var.db_project
    Owner       = var.owner
    Team        = var.team
    Environment = var.environment
  }
}

I was able to get it to work with the below in the aws_instance resource. just not using the local option.

  user_data_base64 = base64encode(templatefile(
      "${path.module}/../../../dev/services/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl",
      merge(
        var.user_data_vars,
        {
          datacenter            = var.short_region
          environment           = var.environment
          environment_lower     = var.environment_lower
          salt_master           = var.salt_master
          hostname              = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}.${var.domain}"
          db_name               = var.db_name
          db_version            = var.db_version
          db_dc_name            = var.db_dc_name
        },
      )
    )
  )

seems that for whatever reason, after making these template changes, my template is not running. i turned on debug but not seeing anything useful to tell me why it’s not being called.

current code … does user_data_base64 not get called inside of resource “aws_instance” “db_instance”?

resource "aws_instance" "db_instance" {
  count     = var.db_server_count
  subnet_id = element(sort(data.aws_subnet_ids.APP_subnet.ids), count.index)
  key_name               = var.aws_key_name
  ami                    = data.aws_ami.db_ami.id
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.db_sec_grp.id]
  
  lifecycle {
    ignore_changes = [ami, instance_type, user_data_base64]
  }

  user_data_base64 = base64encode(templatefile(
      "${path.module}/../../../dev/services/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl",
      merge(
        var.user_data_vars,
        {
          datacenter            = var.short_region
          environment           = var.environment
          environment_lower     = var.environment_lower
          salt_master           = var.salt_master
          hostname              = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}.${var.domain}"
          db_name               = var.db_name
          db_version            = var.db_version
          db_dc_name            = var.db_dc_name
        },
      )
    )
  )

  dynamic "ebs_block_device" {
    for_each = var.ebs_block_device
    content {
      device_name = ebs_block_device.value.device_name
      volume_size = lookup(ebs_block_device.value, "volume_size", null)
      volume_type = lookup(ebs_block_device.value, "volume_type", null)
      iops = lookup(ebs_block_device.value, "iops", null)
      throughput = lookup(ebs_block_device.value, "throughput", null)
    }
  }

  tags = {
    Name        = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}"
    CNAME       = "${var.short_env}-${var.short_region}-${var.db_name}-${count.index + 1}.${var.domain}."
    Application = var.app_name
    Role        = var.db_type
    Project     = var.db_project
    Owner       = var.owner
    Team        = var.team
    Environment = var.environment
  }
}

just realized that the templateFile is not working when using instance type of t2.micro. any reason why?

Hi @kstephenson,

First I want to apologize that I didn’t notice you were using count.index as part of the variables there. As you found out, in that case it’s typically easiest to write the templatefile expression inline in the resource block since then count.index will be available for you.

Although I think what you did here makes the most sense for the current situation, I wanted to note that there is a way to get a result corresponding with your old data "template_file" block with count set, like this:

locals {
  dba_user_data = [
    for i in range(var.db_server_count) : templatefile(
      "${path.module}/../../../dev/services/${var.db_type}-cluster/templates/${var.db_type}-user-data.tpl",
      # ...
    )
  ]
}

This for expression using the range function achieves a similar result as a data block with count set, allowing you to use i inside the expression (in this case, in the template variables) in the same way as you might use count.index in a resource block.

You can then access this from inside a resource block that also has count set to var.db_server_count, using an expression like local.dba_user_data[count.index].


With that said, your newer problem with the template apparently not being rendered is interesting. I don’t know a reason why Terraform’s handling of that expression would vary depending on the instance type, but there might be something happening in the AWS provider that I’m not familiar with.

In order to make it easier to see what’s happening, it might be better to switch to using user_data instead of user_data_base64 and remote the base64encode call, and then you can hopefully see in the plan what the result of template rendering is. Then you can check whether it matches what you expected. You can always switch it back to using base64 once you’ve got it working if you like, though unless it’s a big object or contains non-text data I would typically suggest using the plain user_data so that plans will be easier to review.

it was working but once i switched to instance type t2.micro … it doesn’t work any longer. is there some kind of restriction?

got it … wasted like 5 hours of troubleshooting but it now works. i guess t2.micro doesn’t work for some reason but t3.micro worked great.