Building string array to pass via AWS user_data

Hi,

I am trying to pass the list of private_ip for a collection of compute nodes to the head node in a computational cluster but not getting much joy so far.

resource "aws_instance" "head_node" {
  ami             = "ami-0a269ca7cc3e3beff"
  instance_type   = "t3.large"
  security_groups = [aws_security_group.head_node_sg.name]
  key_name        = "testssh"
  user_data       = templatefile("${path.module}/head_node_setup.sh", {
    efs_hostname = aws_efs_mount_target.cluster_efs_mt["ca-central-1a"].dns_name
    _cnodes_ip = []
    for ip in aws_instance.compute_nodes.private_ip:
      _cnodes_ip.append(ip)
    cnodes_ip    = tostring(_cnodes_ip)
  })

I am aiming for a space separated string I can tokenize in my script but I can grok other representation I guess.

Cheers

Hi @nyue,

Terraform uses a declarative language based on expressions rather than an imperative language based on conditional branches and loops, so to build that cnodes_ip attribute requires writing a single expression to construct the value all at once. The general answer to building collections from other collections is for expressions:

resource "aws_instance" "head_node" {
  ami             = "ami-0a269ca7cc3e3beff"
  instance_type   = "t3.large"
  security_groups = [aws_security_group.head_node_sg.name]
  key_name        = "testssh"
  user_data       = templatefile("${path.module}/head_node_setup.sh", {
    efs_hostname = aws_efs_mount_target.cluster_efs_mt["ca-central-1a"].dns_name
    cnodes_ip = [
      for node in aws_instance.compute_nodes : node.private_ip
    ]
  })
}

Depending on your experiences with other languages, you might recognize this as being similar to list comprehensions in Python, or the Array.map method in JavaScript, or the Array.collect method in Ruby.

Inside the head_node_setup.sh template itself you can further transform this into a space-separated string to include in the script with ${join(" ", cnodes_ip)}. It’s also valid to use join directly in the user_data argument, but because the concatenation with spaces is happening only as an implementation detail of the template itself (future evolution of that template might call for it to be serialized to a string some other way) I’d recommend using join in the template for separation of concerns.

Thanks @apparentlymart, the join works.

Given that it is a list, I was hoping to using formatlist to generate commands from it but validate fails (the formatlist documentation showed an example using list as input)

echo ${join(" ", cnodes_ip)} >/tmp/cnodes.txt
echo ${formatlist("ssh-keyscan -H %s >> ~/.ssh/known_hosts", cnodes_ip)} >     /tmp/keyscan.sh
chmod +x /tmp/keyscan.sh
runuser -l ec2-user -c '/tmp/keyscan.sh'

with the following message

Error: Error in function call

on main.tf line 144, in resource “aws_instance” “head_node”:
144: user_data = templatefile("${path.module}/head_node_setup.sh", {
145:
146:
147:
148:
149:
|----------------
| path.module is “.”

Call to function “templatefile” failed: ./head_node_setup.sh:16,8-72: Invalid
template interpolation value; Cannot include the given value in a string
template: string required…

Hi @nyue,

The result of formatlist is itself a list, so it can’t be inserted directly into a string like that: you’ll need to do an additional transformation to decide how to convert the list to a string first.

It looks like your goal is to run ssh-keyscan once for each of the hosts in your list. If you’re going to be writing the list of IP addresses out to that separate file anyway then I’d personally prefer to read that file in the shell script, because I think it’s easier to read code that works with data rather than code that generates more code. For example (assuming bash is the shell):

while read line; do
  for addr in $line; do
    ssh-keyscan -H "${addr}" >>"~/.ssh/known_hosts"
  done
done

If the above were in a file named /tmp/keyscan.sh then you could run it like this:

bash "/tmp/keyscan.sh" <"/tmp/cnodes.txt"

(note that the ${addr} in the above is a bash environment variable interpolation, not a Terraform template interpolation, even though both use the same syntax. I’m writing the above assuming it will be in a separate file not created dynamically using Terraform, but if you wanted to create it using your script templated from Terraform then you’d need to escape it as $${addr} to ensure that Terraform will ignore it and let the shell interpret it instead.)

If you’re keen on templating that out with Terraform itself then I’d suggest using a template for sequence to repeat a line for each element in cnodes_ip, like this:

%{ for addr in cnodes_ip ~}
ssh-keyscan -H '${addr}' >>"~/.ssh/known_hosts"
%{ endfor }

Writing a Terraform template to generate a bash command that generates another bash script containing the result of another Terraform template is a pretty complicated mess that I’m not sure exactly how to write robustly, because you’d need to contend with three levels of language embedded inside other language, each of which requiring some sort of escaping. The only approach that comes to my mind right now is to nest one template inside another and use replace to handle the escaping:

echo '${replace(<<EOT
%{ for addr in cnodes_ip ~}
ssh-keyscan -H '${addr}' >>"~/.ssh/known_hosts"
%{ endfor }
EOT
, "'", "\\'")}' >/tmp/keyscan.sh
chmod +x /tmp/keyscan.sh
runuser -l ec2-user -c '/tmp/keyscan.sh'

I would not want to be the future maintainer of something as inscrutable as that, though. Maybe it’s possible to do it all in one shot directly inside the user_data script like this:

for addr in ${ join(" ", var.cnodes_ip) }; do
  ssh-keyscan -H "$${addr}" >>"~/.ssh/known_hosts"
done

In the above I’ve tried to just insert the IP addresses directly into the for statement. I think rendering the above template would produce a result like this, which might work for what you need:

for addr in 10.1.2.1 10.1.2.2 10.1.2.3; do
  ssh-keyscan -H "${addr}" >>"~/.ssh/known_hosts"
done

Hopefully something in this comment is useful! This particular requirement is a bit messy to deal with because of how many layers are involved, so honestly my instinct here would be to try to change the problem to something that doesn’t require so much bespoke setup on first instance boot, but you know better than I do what your constraints are! :wink: