AWS User Data with Multiple Files using Templatefile

Terraform
3

Thanks @apparentlymart. In short, I am trying to pass multiple setup scripts to the EC2 instance and interpolating variables into them at creation time. Per the documentation, I should be using templatefile instead of template_file. Both of these do not support merging files. I have to leverage cloudinit_config (or template_cloudinit_config) along with template_file to accomplish this. I am trying to find a way to do this using templatefile instead of template_file since I only know some variables at EC2 creation time.

I was able to make this work by creating a merge_files module (below) and just calling that, but it is a bit hacky.

// modules/helper/merge_files/main.tf

variable "files" {
  type = list(string)
}

locals {
  files = {
    for index, file in var.files: index => file
  }
}

data "local_file" "files" {
  for_each = local.files
  filename = each.value
}

data "template_cloudinit_config" "merged" {
  gzip          = false
  base64_encode = false

  dynamic "part" {
    for_each = data.local_file.files

    content {
      filename     = "${part.key}_${basename(part.value.filename)}"
      content_type = "text/x-shellscript" 
      content      = "${part.value.content}"
    }
  }
}

resource "local_file" "merged" {
  content = data.template_cloudinit_config.merged.rendered
  filename = "${path.root}/.terraform/tmp/merged_${timestamp()}.txt"
}

output "filename" {
  value = local_file.merged.filename
}

// main.tf

module "merged_file" {
  source = "./modules/helper/merge_files"
  files = [
    "${path.root}/scripts/common.sh",
    "${path.module}/scripts/configure.sh"
  ]
}

resource "aws_instance" "server" {
  for_each = var.servers

  ami = each.value.ami_id
  instance_type = each.value.instance_type
  availability_zone = each.value.availability_zone
  subnet_id = each.value.subnet_id
  vpc_security_group_ids = each.value.security_group_ids
  key_name = var.key_name

  user_data = templatefile(module.merged_file.filename, { 
    hostname = "${each.value.environment}-server-${each.value.index}.${each.value.domain}"
  })

  tags = {
    Name = "${each.value.environment}-server-${each.value.index}"
  }
}

Per your reply here, I see that you recommend not using user_data to pass in setup scripts, but rather to use Packer to build those into the base image initially so that you can just call them via user_data instead. I do like this idea, and currently use Packer now for the images, but it can be quite a bit inefficient to have to rebuild them for every setup script change, especially if there are multiple layers of image dependencies (e.g. base-image which is used for service images, etc). I guess a possible solution would be to deploy shared scripts to a network filesystem (e.g. EFS) and just mount them on EC2 creation to gain both immutable infrastructure and access to updated scripts at the same time.