Select three most recent AMI IDs

I have a not too similar situation. Relatively new to terraform and have a requirement to loop through a list of ami-ids in a source account to pick out the most recent 3 ami-ids to create an ssm-parameter with 3 versions in a destination account. how could i achieve that? i keep getting the below error:

│ Error: Invalid index
│
│   on main.tf line 27, in resource "aws_ssm_parameter" "this":
│   27:   value       = data.aws_ami_ids.this[each.value].ids
│     ├────────────────
│     │ data.aws_ami_ids.this is object with 1 attribute "AMZNX-2"
│     │ each.value is "aerospike-amazonlinux2*"
│
│ The given key does not identify an element in this collection value.

variables.tf

variable "ami_map" {
  type = map(string)
  default = {
    "AMZNX-2" : "aerospike-amazonlinux2*"
  }
}

main.tf

data "aws_ami_ids" "this" {
  for_each = var.ami_map

  provider = aws.source
  sort_ascending = true
  owners      = [var.owner_id]
  filter {
    name   = "name"
    values = [each.value] # regexp
  }
  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_ssm_parameter" "this" {
  for_each    = toset(keys(var.ami_map))
  
  provider    = aws.destination
  type        = "String"
  name        = "/dummy/ami/${each.key}/ami_id" # ssm path
  value       = data.aws_ami_ids.this[each.value].ids
  overwrite = true
  }
}

Hi @nc237,

I think the error you’ve seen here is not directly related to your goal but your configuration has a few different problems and so Terraform is detecting one problem and then being blocked from detecting the others, so the error message is about a different problem than your question is about.

With that said, let’s return to the original problem statement and put your current configuration aside for the moment. It seems like this problem is broken into two parts:

  • Find the ids of the most recent three AMIs matching a particular name pattern, given as a regex.
  • For each set of IDs found, write the IDs into an SSM parameter.

When I compare what I understand of the problem statement to your example I see a mismatch, so before continuing I want to make sure I understand correctly your goal. Your resource "aws_ssm_parameter" "this" block seems to be declaring a parameter of type "String", but your problem statement suggests that there can be up to three IDs for each parameter which would suggest that StringList would be the more appropriate type.

Do you intend to create a single /dummy/ami/AMZNX-2/ami_id parameter which contains a list of up to three IDs, or do you instead intend to create up to three parameters per element of var.ami_map? If the latter, what’s the naming scheme to ensure that each of the three AMIs has a unique parameter name?

Hello @apparentlymart, appreciate you getting back to me. The goal is to create a /dummy/ami/AMZNX-2/ami_id parameter with a list of 3 IDs (versions) for a an AMI of type Amazon Linux2 for example. I know my code is a little all over the place :slight_smile: need some help refactoring the code to be able to grab the latest 3 ami-ids and use that to create an ssm parameter with all 3 IDs…if that is possible. Thanks in advance

Thanks for the clarification, @nc237!

Based on your latest comment I expect that you need a StringList parameter rather than String parameter, so I’m going to write it that way. That detail is not super important to this example so hopefully you can adjust that part to be a different way if you need to.

Otherwise, I think we can break this down into a few different steps. The first one you seem to already have solved, which is to retrieve the IDs of AMIs matching your given name pattern:

data "aws_ami_ids" "this" {
  provider = aws.source
  for_each = var.ami_map

  sort_ascending = true
  owners         = [var.owner_id]

  filter {
    name   = "name"
    values = [each.value] # regexp
  }
  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

This is the same as what you already provided. I’m not an expert on this particular data source so I’m not sure if there’s some limit on the number of items it can possibly return so that it won’t get gradually slower over time as you add more AMIs. The main assumption here though is that if it does have a limit then that limit is greater than three, so we’ll always get at least enough information to deal with the next step.

The aws_ami_ids data source has sort_ascending = true and so I assume that means that the first three items in its result will be the most recent three, and so the next step is to truncate these lists to at most three items:

locals {
  amis = tomap({
    for k, q in data.aws_ami_ids.this : k => chunklist(q.ids, 3)[0]
  })
}

This expression has a few tricky parts to it, so I’ll break it down into smaller pieces:

  • The { for ... in ... : ... } part is a for expression which constructs one data structure from the elements of another. In this case, the source collection is the map of instances of data.aws_ami_ids.this, and so k is the instance key (always matching a key from var.ami_map) and q – short for “query” – is the object representing the corresponding instance of the data resource.

    Terraform evaluates the expressions on the right hand side of the colon for each element of data.aws_ami_ids.this, so the result has the same number of elements as the input.

  • The chunklist function here is a bit cheeky: it’s asking Terraform to split the given list of IDs into chunks of at most three elements, and then taking only the first chunk and discarding the rest.

    I did it this way because chunklist automatically handles the case where there are fewer than three elements in the list, whereas it would be more finicky to handle that with other approaches. However, in case that seems “too clever” here’s another expression that is more verbose but perhaps clearer about what it’s doing:

    slice(q.ids, 0, min(length(q.ids), 3))
    
  • Finally, the surrounding tomap is optional but I like to include this to be explicit that I’m intending this result to be used as a map rather than as an object. A { for ... } expression generates an object by default because it’s the more general kind, but object types in Terraform typically represent a fixed set of attributes whereas a map represents an arbitrary set of keys decided dynamically.

local.amis should therefore be a map from each of the keys in your initial var.ami_map to a list of at most three AMI ids.

The final step is to write these in to SSM using aws_ssm_parameter. The documentation for that resource type is unclear on how exactly you’re supposed to submit a StringList, but I’m guessing from the corresponding CLI documentation that the provider is expecting value to be a string containing comma-separated values, and so I’ve written the following under that assumption:

resource "aws_ssm_parameter" "this" {
  provider = aws.destination
  for_each = local.amis
  
  type      = "StringList"
  name      = "/dummy/ami/${each.key}/ami_id"
  value     = join(",", each.value)
}

The local.amis value is a map with one element per SSM parameter you want to create, so it’s suitable for direct use in for_each without any further transformation. The only remaining complications then are to interpolate each.key into the name and to transform the list of separate ID strings from each.value into a single string with commas separating the items.

I don’t have active AWS credentials at the moment so I’ve just written this out directly into the comment box without testing it, and so I’m sure I’ve made at least one typo somewhere. If you try this and see any errors that you aren’t sure how to resolve, please let me know and I’ll try to correct myself!

Hello, relatively new to terraform and have a requirement to loop through a list of ami-ids in a source account to pick out the most recent 3 ami-ids to create an ssm-parameter with 3 versions in a destination account. how could i achieve that? i keep getting the below error:

│ Error: Invalid index

│ on main.tf line 27, in resource “aws_ssm_parameter” “this”:
│ 27: value = data.aws_ami_ids.this[each.value].ids
│ ├────────────────
│ │ data.aws_ami_ids.this is object with 1 attribute “AMZNX-2”
│ │ each.value is “aerospike-amazonlinux2*”

│ The given key does not identify an element in this collection value.

variables.tf

variable “ami_map” {
type = map(string)
default = {
“AMZNX-2” : “aerospike-amazonlinux2*”
}
}

main.tf

data “aws_ami_ids” “this” {
for_each = var.ami_map

provider = aws.source
sort_ascending = true
owners = [var.owner_id]
filter {
name = “name”
values = [each.value] # regexp
}
filter {
name = “virtualization-type”
values = [“hvm”]
}
}

resource “aws_ssm_parameter” “this” {
for_each = toset(keys(var.ami_map))

provider = aws.destination
type = “String”
name = “/dummy/ami/${each.key}/ami_id” # ssm path
value = data.aws_ami_ids.this[each.value].ids
overwrite = true
}
}

Hello again @apparentlymart, thanks for the input. Worked like a charm, and seems the locals block with its expression was the game changer here. However, it seems i did not get the output i was hoping, but that should be entirely my fault.

In ssm, this is what the parameter and its value look like:
name: /dummy/path/AMZNX-2/ami_id
value: ami-09829bcad6524852d,ami-070636a6009852298,ami-0f01067242c915655

I just realized i was rather trying to ensure each id is a version of that parameter. for instance in the ssm parameter console under the history tab, the 3 ids should show up as 3 versions of that parameter…if that makes sense

Hi @nc237,

I don’t know what 4014711211 represents, but since this is a plural data source I can’t imagine anything useful it could be writing into id and so I expect this is probably just some random garbage the provider is writing in there for the sake of generating some sort of ID, rather than it being something meaningful.

The actual results of data lookups don’t typically appear directly in the plan output, but I would expect you to be able to see plans to create one or more instances of aws_ssm_parameter.this where the name and value attributes will show you (indirectly) the results.

Yep. It wasn’t an issue. Thanks again.

I think you might have edited this comment while I was replying to it, so I’m going to reply again to the current content. :grinning:

I think what you are describing now is beyond my (very limited) knowledge of AWS SSM. So far this question has broadly been about Terraform language features in general and the fact that we’re talking about AMIs and SSM parameters has been a relatively-unimportant detail, but if we start talking about multipple versions of a parameter then I think that will require using the AWS provider features in a different way, but I don’t know enough to say what that is.

Do you already know of a way to declare separate versions of an SSM parameter via the AWS provider? If you happen to already know how to do it manually (in a resource with hard-coded arguments) and can show an example then I’d be happy to try to adapt it into something more dynamic.

Yeah I edited the comment my bad :slight_smile: At the moment, i am not aware of a way other than overwriting the parameter with new values. It seems SSM stores the previous version upon each update. You are a genius! Thank you for the helpful tips!

Oh, interesting!

So perhaps then the refined problem statement is for your Terraform configuration to write only the latest AMI into parameter store, and then over time parameter store will accumulate a history of previous values automatically even though Terraform itself isn’t aware of it.

I think the same overall approach I showed still applies in that case, but you can probably cut out the intermediate chunklist step by using aws_ami instead of aws_ami_ids since you’ll only need the one latest AMI now, as long as you’re guaranteed that there will always be at least one matching AMI. (If there isn’t then aws_ami will fail whereas I think aws_ami_ids will return an empty list.)

That was exactly my thought! However, I was wondering if there was a way to make our code update that parameter 3 times? That way SSM retains the value upon each update such that it can keep all 3 versions?

It sounds like you want to just create an initial history of the most recent three right now and then manage it in the normal way with Terraform moving forward. Is that right?

If so, I would probably do that by performing the initial writes of the third oldest and second oldest AMIs just using the AWS CLI directly, to populate the first two entries in the history, and then apply your Terraform configuration to overwrite with the current latest.

There probably are some ways to convince Terraform to write all three in a particular order but because ordering in Terraform is defined only for whole resource blocks rather than individual instances you would need three separate resource blocks with dependencies between them to force the correct order of writes. That seems like a lot of fuss just for a one-off import, and I expect it would also be difficult to clean up afterwards because you would need to ask Terraform to “forget” the older two so it won’t try to delete the SSM Parameter to reconcile the new desired state.

Managing this “properly” with Terraform would be possible in principle if the AWS provider was modelling individual versions as a resource type, rather than the parameter as a whole, but I suspect the underlying API also isn’t really suited to that usage pattern. (It would require an API action to insert a particular value into the history, rather than only to create a new latest version.)

Thank you for all the valuable insight!