Use locals to loop aws_instance through 3 subnets

Hi all,

I am trying to set up a locals resource block with 3 already created subnets from one of our AWS accounts so I can deploy a large number of webservers to an unmanaged infrastructure (as in not created by terraform).

Now I could create 3 aws_instance resource blocks and be done with it, but if I do believe I should be able to list the IDs some how and have the one awsinstance block which then loops through each subnet (az) so that way they are in 3 different AZs too then.

Here is me code:

locals {
  # Subnet IDs to loop through
  subnet_ids = [{
    "subnet-0ad390442883258a9",
    "subnet-0b4ef4bbad2c5d7e8",
    "subnet-0519c71d2872bf471"
  }]
}

resource "aws_instance" "ttt_iis" {
  ami           = var.ami
  instance_type = "m5.large"
  key_name      = "Infrastructure Build"
  user_data = templatefile("psdata.tpl", {machine_name = var.machine_name})
  iam_instance_profile = var.instance_profile
  subnet_id = locals.subnet_ids
  # subnet_id     = "subnet-06209c9078bca40c2" ## Tier 2 AZ2
  # when moving to prod, uncomment top 2, remove bottom 2.
  vpc_security_group_ids     = [
    # "sg-0583d6aeafc4304b3",
    # "sg-0b32797bbedbcb33d"
    "sg-0b0aaf8750fa3d6b9",
    "sg-0eba6606442768b19"
  ]

  root_block_device {
    volume_size = 50
    encrypted = true
  }

  tags = {
    Name = var.machine_name
  }  
}

I was just wondering if someone could help, advice or point me where I am going wrong.

Kindest Regards,

James

Hi @jammyshaw,

You should be able to get a result like you’re asking about using resource for_each, which allows us to systematically declare multiple instances of a resource using a single resource block.

When using for_each like this, we need to give Terraform a unique string key with which to track each of the instances between runs. Since your three subnet IDs are hard-coded in your example, you could in principle use those IDs directly as those keys, but it’s more common for IDs like those to be dynamically chosen and thus not suitable for for_each, and so I’m not going to show an example of that just because I don’t want to show you a seeminly-straightforward example that would work in this specific situation but would not generalize to more common situations.

Instead, I’d suggest to change the local value to be a map where the keys are fixed strings that won’t change if you were to recreate the subnets in future, and the values are the subnet ids. For the sake of this example, I’m going to assume AWS availability zone names are a suitable unchanging unique key, though that would not be true if you were to create more than one subnet in the same AZ; in that case, you can choose other keys that make sense for whatever you’re trying to model.

locals {
  subnet_ids = tomap({
    "us-east-1a" = "subnet-0ad390442883258a9"
    "us-east-1b" = "subnet-0b4ef4bbad2c5d7e8"
    "us-east-1c" = "subnet-0519c71d2872bf471"
  })
}

resource "aws_instance" "example" {
  for_each = local.subnet_ids

  instance_type = "m5.large"
  subnet_id     = each.value
  # ...
}

The above configuration declares that there should be one instance of aws_instance.example for each element of local.subnet_ids, with each one assigned an address based on the keys in that map:

  • aws_instance.example["us-east-1a"]
  • aws_instance.example["us-east-1b"]
  • aws_instance.example["us-east-1c"]

Because I assigned each.value to subnet_id in the configuration, each of these instances will have its own different value of that argument, which will be the value from the map declared above.

Because Terraform will be tracking these by the map keys, you can later add new entries to the map in order to (indirectly) declare new instances, or reassign the subnet ID for an existing key to (indirectly) tell Terraform to replace that particular instance.


Incidentally, while it is valid and reasonable to declare multiple AWS instances directly like this, for this particular situation where it seems like your goal is to spread workload across multiple AZs it might be worth considering using aws_autoscaling_group instead, which makes EC2 itself responsible for managing the multiple instances and thus allows the remote system to respond dynamically to problems such as instance failures and AZ-level outages.

If you were to use an autoscaling group instead of multiple directly-managed EC2 instances here, you’d specify the full set of subnets all together on a single autoscaling group object, similar to what you originally attempted with aws_instance, and then from Terraform’s perspective the “group” behaves as a single object whereas the individual instances are created dynamically by EC2 autoscaling.

This also allows you to separate the number of instances you need from the number of availability zones to spread them over: autoscaling will either choose a subset of the subnet IDs you specified (if the desired count is smaller than the subnet count) or create more than one instance in the same subnet (if the desired count is greater than the subnet count), and thus you can potentially scale up or down your number of instances without also creating or destroying subnets.

To do that with the local value I used in the above example, you could use values to take just the subnet ids again, or you could revert to your original idea of just making a single set/list of ids and passing that directly. In both cases, the subnet IDs need to be assigned to the (rather oddly-named) vpc_zone_identifier argument:

  vpc_zone_identifier = values(local.subnet_ids)

Oh this all makes me want to cry, why did I move into devops/infrastructure to learn automations! :rofl:

Seriously though, I really appreciate the time and effort you went into explaining it all, I’ll try and soak it all in and get something working.

Thank you once again.

Unfortunately, I couldn’t get this working…

I assume I will need 3 resource blocks for aws_instance? I am pretty much very new so still trying to learn how everything comes to gether etc.

Also, my blooming dev box has corrupted, so I can not get on to it at this current time to grab the error messages I am receiving.

What I should also mention, is within my code, I do use a module block as well as a few further SSM resource blocks which will add all machines created to the current Managed AD. In the module block, this controls things such as AWS Tag name etc. and will also use user data to copy the AWS Tag name to set as hostname ont he machine. Once I get my box sorted, I’ll supply errors/further code to see if I can get further advice on getting this working :slight_smile:

Thank you very much for your help so far.

Ok, fixed the dev box. My code is split through 3 files, there is a main.tf varaibles.tf (currently variables isn’t utilised with what I am currently trying to achieve so won’t post its content) and then ws-deploy.tf

The main is (it does have other bits etc. Such as SSM, aws providers etc but they don’t interfere with what I am doing so I won’t paste then to try and keep the post as short as possible):

main.tf

#Module block for aws_instance resource in ws-deploy.tf 

module "create_ttt_az1_servers" {
  count = var.iis_count_az1
  source = "./create_webserver"
  machine_name = "TTT-Web-T3-${count.index + 16}" ## Start hostname at which ever given number.
  ami = var.windows2019_base_ami_id
  (commented out) subnet_ids = var.subnet_ids
  instance_profile = aws_iam_instance_profile.instance_profile_adwriter.name
  ssm_document = aws_ssm_document.ttt_joinad_doc.name
}

ws-deploy.tf

variable machine_name {}
variable ami {}
variable instance_profile {}
variable ssm_document {}

locals {
  (commented out) Subnet IDs to loop through
  subnet_ids = [{
    "subnet-0ad390442883258a9",
    "subnet-0b4ef4bbad2c5d7e8",
    "subnet-0519c71d2872bf471"
  }]
}

resource "aws_instance" "ttt_iis" {
  ami           = var.ami
  instance_type = "m5.large"
  key_name      = "Infrastructure Build"
  user_data = templatefile("psdata.tpl", {machine_name = var.machine_name})
  iam_instance_profile = var.instance_profile
  subnet_id = locals.subnet_ids
  (commented out) subnet_id     = "subnet-0519c71d2872bf471" ## change when deploying to prod to subnet AZ1 TTT.
  (commented out)when moving to prod, uncomment the 2 SG as they are TTT IIIS rule/role groups.
  vpc_security_group_ids     = [
    # "sg-0583d6aeafc4304b3",
    # "sg-0b32797bbedbcb33d"
    "sg-0b0aaf8750fa3d6b9",
    "sg-0eba6606442768b19"
  ]

  root_block_device {
    volume_size = 50
    encrypted = true
  }

  tags = {
    Name = var.machine_name
  }  
}

resource "aws_ssm_association" "ttt_adwriter" {
	name = var.ssm_document
	instance_id = aws_instance.ttt_iis.id
}

Now, when I try altering my code to what you have put, I get the following:

$ terraform1.0 plan
Error: Missing resource instance key

  on create_webserver/ws-deploy.tf line 52, in "aws_ssm_association" "ttt_adwriter":
  52:         instance_id = aws_instance.ttt_iis.id

Because aws_instance.ttt_iis as "for_each" set, its attributes must
be accessed on specific instances.

For example, to correlate with indices of a referring resource, use:
    aws_instance.ttt_iis[each.key]

$ terraform1.0 plan
Error: Reference to "each" in context without for_each

  on create_webserver/ws-deploy.tf line 52, in "aws_ssm_association" "ttt_adwriter":
  52:         instance_id = aws_isstance.ttt_iis[each.key].id

The "each" object can be used only in "module" or "resource" blocks,
and only when the "for_each" argument is set.

I am kind of assumign I need to do something similar to below, but again being quite new to terraform, my knowledge doesn’t really expand that far so I am not sure where to go from here, considering the each object is in a resource block (resource ssm assosiciation).

The above configuration declares that there should be one instance of aws_instance.example for each element of local.subnet_ids , with each one assigned an address based on the keys in that map:

  • aws_instance.example["us-east-1a"]
  • aws_instance.example["us-east-1b"]
  • aws_instance.example["us-east-1c"]

Now, I know can create 3 aws_instance resource blocks and add the subnet_id in manually and it works that way, but that then screws my machine_name variable up in my module block, because I need these to start at 16 and take numbers from there, having 3 resource blocks just means they are all using the same number (I.E. I’ll create 3 instances and they will all be like TTT-Web-T3-16, not 16,17 & 18) and this is why I am trying to create a loop for the subnets instead, so its just one aws_instance resource that can create instances in each subnet.

I have probably complicated this more than I should of, but for me is a learning experience and getting it working is definitely the main thing, and then cleaning it up can come second.

Kind Regards,

James

Hi @jammyshaw,

The error messages you shared here are the result of your resources now disagreeing with one another about how many of each resource type you’re declaring.

Specifically, if you declare multiple aws_instance.ttt_iis then you need to also declare multiple aws_ssm_association, assuming that you want to create one SSM association for each of your instances.

The structure for something like that would be the following:

resource "aws_instance" "ttt_iis" {
  # One instance per subnet id
  for_each = local.subnet_ids

  # ...
}

resource "aws_ssm_association" "ttt_adwriter" {
  # One SSM association per instance
  for_each = aws_instance.ttt_iis

  name        = var.ssm_document
  instance_id = each.value.id
}

This is an example of chaining for_each between resources: a resource with for_each set appears in expressions as a map of objects, and that’s compatible with the expectations of for_each on other resources and so in this case we can write for_each = aws_instance.ttt_iis to declare one SSM association for each instance, where the instance keys will all correlate with the aws_instance.ttt_iis ones. each.value inside that block is therefore an object representing one of the instances, and thus we can write each.value.id to get the ID to assign to instance_id.

As previously. Thank you so much for your support and assistant, I never knew for_each could be chained between resources which is going to be really useful for future projects now.

Amazing community and people helping out. So, I did implemented your new code and did run into a problem with the locals block. As you can see here:

Ofcourse, this was quite easy to debug considering it gives me the exact problem at hand, and all I did was swap the subnets and az around like so:

locals {
subnet_ids = tomap({
“subnet-0519c71d2872bf471” = “eu-west-2a”
“subnet-0ad390442883258a9” = “eu-west-2b”
“subnet-0b4ef4bbad2c5d7e8” = “eu-west-2c”
})
}

You have been a real superstar in helping me getting this nipped, I can’t thank you enough!! :slight_smile: :slight_smile:

Just a informational point to - TTT = The tools trade is a business adventure I am working on one my own aws account, dont want people thinking I am exposing others. :slight_smile:

Have a great weekend,

James

Well, I was hoping I didn’t have to bother or take anyones time again. Unfortunately, after further testing and deploying, I noticed that my machine names aren’t working exactly how they should.

Going back to my module block, I use both the count and count.index within here

#Module block for aws_instance resource in ws-deploy.tf 
module "create_ttt_az1_servers" {
  count = var.iis_count ## Change count in variables.tf
  source = "./create_webserver"
  machine_name = "TTT-Web-T3-${count.index + 16}

Your changes are exactly what I needed to be honest, as now my aws_instance does loop through the subnets listed, now the only thing is, each instance doesn’t start its machine_name with a unique number, previously it was start whatever + # was at, then go from there I.E 16, 17, 18 and so on. Now, it does somewhat follow this still but it does it in 3s, so the first 3 instances created are TTT-Web-T3-16, then the next 3 would be 17.

In the variables file I have:

variable "iis_count" {
  type = string
  # Edit default to required number needed.
  default = "2"
}

And under my ws-deploy.tf file, we have tags:

resource "aws_instance" "ttt_iis" {
  for_each      = local.subnet_ids # for_each acts as the start of the loop for subnets.
  ami           = var.ami
  instance_type = "m5.large"
  key_name      = "Infrastructure Build"

  tags = {
    Name = var.machine_name # variable for machine name, can be changed in main.tf
  }  
}

If there is a quick and easy way to stop this from happening, I would appreciate a little more assistance, I am currently investigating online so if I do find a fix before anyones assistance I’ll be sure to add here so nobody wastes time :slight_smile:

For this one (the InvalidSubnetID.NotFound error) it seems like you used subnet_id = each.key and thus ended up assigning the availability zone to the subnet argument, instead of the subnet id.

The adjustment I’d suggest here is to write subnet_id = each.value instead, so that you can keep the map keyed by the unchanging availability zones, but use the remote-system-assigned subnet ID (which would change if you ever replaced any of those subnets) to populate subnet_id, as the remote system expects.

Hi @jammyshaw,

Usually when declaring multiple instances using for_each like this we want to avoid including sequence-significant identifiers in the configuration, because otherwise adding and removing elements from the for_each map would cause those numbers to change.

While for a Name tag in particular it’s not a huge problem for them to get renumbered later on, that tends to cause a confusing plan and changes identities that humans might be accustomed to relying on in the admin console.

With that said then, I’d recommend one of the following alternative strategies instead:

  • Let all of the instances of this group just have identical Name tags. EC2 doesn’t require that tag names be unique, and it seems like you intend for these instances to all be fungible anyway, so perhaps it makes sense for them to all have identical tags.
  • Alternatively, if there’s something else in your workflow which requires unique name tags, use a construction like Name = "${var.machine_name}-${each.key}" so that they are distinguished by a meaningful characteristic (which AZ they are deployed in) rather than by a meaningless number.

There are ways to get numeric indices back with for_each when you really need them, but it gets pretty awkward and creates the disadvantages I described above, so I’d personally avoid it unless there’s no alternative.

Hi,

As we do need unique Names per instance for auditing purposes and to easily track each system, creating a unique AWS Name was really the only solution I could come up with, as this allows me to utilise powershell through a user data template to then copy the AWS Name and set it as the system hostname to.

Being quite new to terraform, I never knew there was a constraint with for_each and count until you described so this kind of throws a spanner in to my work. If it is possible to get number indices working with the for_each, once these instances are deployed. There is no plan to make any changes to the resources/terraform code.

Thank you for all your support so far.

Kind Regards,

James

Number based naming, while understandable from a human perspective, actually have a number of issues. For example if you have server1 to server8 and then one is stopped (say server5) and replaced would that new server be server9 or server5? How would you differentiate between the “old” and “new” server5 in the second case?

If you really do need a unique name it is probably better to create something based on a number of other data points, such as software versions, AMI, IP address, etc. (possibly hashing those in some way) instead of just adding an arbitrary number to the end.

The number based option does still work well in some more human focussed situations, in which case you’d use count rather than for_each. The main requirement is to ensure that changes only happen at the top end of the number range, so you don’t end up with the situation of a “middle” server being removed and all the subsequent ones having to be destroyed & recreated as they are renumbered.

Hi, @apparentlymart & @stuart-c

Ok, we have decided that the numeric indices can be dropped if it does cause more problems than necessary.

I am going to look at some thing a bit more simple such maybe ttt-az1-servera, ttt-az2-servera, ttt-az3-servera, ttt-az1-serverb, ttt-az2-serverb, ttt-az3-serverb.

Its difficult to find solutions to getting unique AWS Names/hostnames, while AWS does provide us with some, they aren’t exactly easy to identify straight away and with Windows only having a 15 character limit, it becomes even more painful.

I guess even if I did something like ttt-az1-servera, ttt-az2-servera, ttt-az3-servera, ttt-az1-serverb, ttt-az2-serverb, ttt-az3-serverb to remove the numbers at the end, I assume I would still need to create a list of some sort from default tags to be able to do a merge in to resource tags?

Sorry for all these different questions/changes, I am not trying to be a pain, just trying to get something working with unique names that I can pass through user data via powershell to the system hostname.

Kindest Regards,

James

EDIT: I am going to leave this, I really do appreciate the support/guidance, I think this has really determined I need to spend more time learning terraform instead of trying to do complicated stuff.