Add key/value to a map (or deep merge)

Terraform
1

Hi,
iterating on local data struct (map of objects):

locals {
  dr_ec2_instances = {
    "bastionaws1"  =   {
      "instance_id_orig"         = "i-00000000000000001"
      "ami_virtualization_type"  = "hvm"
      "ami_architecture"         = "x86_64"
      "ami_ena_support"          = true
      "ami_rbd_device_name"      = "/dev/sda1"
    }
    "bastionaws2" = {
      "instance_id_orig"         = "i-00000000000000002"
      "ami_virtualization_type"  = "hvm"
      "ami_architecture"         = "x86_64"
      "ami_ena_support"          = false
      "ami_rbd_device_name"      = "/dev/sdb1"
    }
  }
}

data "aws_ebs_snapshot" "dr_snap" {
  for_each = {
    for key, val in local.dr_ec2_instances:
    key => val
    if var.dr_env
  }
  provider = aws.alias
  most_recent = true
  owners      = ["self"]

  filter {
    name   = "status"
    values = ["completed"]
  }

  filter {
    name   = "tag:Name"
    values = ["rbd_${each.key}*"]
  }
}

I extract snapshot map of objects like this

(data.aws_ebs_snapshot.dr_snap) = {
  "bastionaws1" = {
	...
    "id" = "snap-0000000000000000a"
    "most_recent" = true
    "snapshot_id" = "snap-0000000000000000a"
    "state" = "completed"
    "tags" = tomap({
	  ...
    })
    "volume_size" = 8
  }
  "bastionaws2" = {
	...
    "id" = "snap-0000000000000000b"
    "most_recent" = true
    "snapshot_id" = "snap-0000000000000000b"
    "state" = "completed"
    "tags" = tomap({
	  ...
    })
    "volume_size" = 8
  }
}

Iterating on snap map of objects I can create base AMI from which to create ec2

resource "aws_ami" "dr" {

  for_each = {
    for key, val in data.aws_ebs_snapshot.dr_snap:
    key => val
    if var.dr_env
  }
 
  provider = aws.alias
 
  name                = "DR Site instance AMI (${each.key})"
  description         = each.value.description
  virtualization_type = "hvm"
  architecture        = "x86_64"
  ena_support         = true
  root_device_name    = "/dev/sda1"
  tags = {
    ...
  }
 
  ebs_block_device {
    device_name  = "/dev/sda1"
    volume_type  = "gp3"
    iops         = 3000
    throughput   = 125
    snapshot_id  = data.aws_ebs_snapshot.dr_snap[each.key].id
    delete_on_termination = true
  }
}

I can’t find a method to add key/value to (data.aws_ebs_snapshot.dr_snap) to customize ami instead of using static data like: virtualization_type, architecture , ena_support, etc.

e.g. merge dr_ec2_instances with (data.aws_ebs_snapshot.dr_snap) using this new data struct in for_each loop in resource "aws_ami" "dr".

Static data impose to divide and repeat the process for omogeneous groups of destination AMI.

2

A workaround could be to set tags to be present in snapshot data source (original volumes snap was taken from propagating tags), but it is not flexible as impose same config instead of a custom config.

Be able to choose custom config on DR site based on dr_ec2_instances is essential also for the next step with additional ebs volume creation (from other snapshot using dr_ebs) to attach volumes to ec2 created.

3

As Terraform today does not support collections deep merge, I used a workaround based on common collection data.

Making keys relations for two or more collections, allow for_each loop based on one of this collection (we could name base collection) and reference to other collections (we could name secondaries collections) using keys (or values) of base collection (or using a funtion on keys/values , e.g. regex) as elements identifiers in secondaries collectios.

Example (dr_ec2_instances can be defined also as variable for different env configurations or as dynamic data using resource or module output)


locals {
  dr_ec2_instances = {
   "bastionaws1"  =   {
      ...
    }
    "bastionaws2" = {
      ...
    }
  }
}

data "aws_ebs_snapshot" "dr_snap" {
  ...
}




with code above will have 2 related collections



 (base collection)
  dr_ec2_instances = {
    "bastionaws1"  =   {
      "instance_id_orig"         = "i-00000000000000001"
      "ami_virtualization_type"  = "hvm"
      "ami_architecture"         = "x86_64"
      "ami_ena_support"          = true
      "ami_rbd_device_name"      = "/dev/sda1"
      "ami_rdb_volume_type"      = "gp3"
      "ami_rdb_iops"             = 3000
      "ami_rdb_throughput"       = 125
    }
    "bastionaws2" = {
      "instance_id_orig"         = "i-00000000000000002"
      "ami_virtualization_type"  = "hvm"
      "ami_architecture"         = "x86_64"
      "ami_ena_support"          = false
      "ami_rbd_device_name"      = "/dev/sdb1"
      "ami_rdb_volume_type"      = "gp3"
      "ami_rdb_iops"             = 3000
      "ami_rdb_throughput"       = 125
    }
  }

(secondary collection)
(data.aws_ebs_snapshot.dr_snap) = {
  "bastionaws1" = {
    ...
    "id"          = "snap-0000000000000000a"
    "state"       = "completed"
    "volume_size" = 8
    ...
  }
  "bastionaws2" = {
    ...
    "id"          = "snap-0000000000000000b"
    "state"       = "completed"
    "volume_size" = 8
    ...
  }
}




We can use both collections without merging anything


resource "aws_ami" "dr" {

  for_each = {
    for key, val in data.aws_ebs_snapshot.dr_snap:     <-- (base collection)
    key => val
    if var.dr_env
  }
 
  provider = aws.alias
 
  name                = "DR Site instance AMI (${each.key})"     <-- (base collection)
  description         = each.value.description                   <-- (base collection)
  virtualization_type = local.dr_ec2_instances[each.key].ami_rdb_volume_type    <-- (secondary collection)
  architecture        = ...    
  ena_support         = ...
  root_device_name    = local.dr_ec2_instances[each.key].ami_rbd_device_name    <-- (secondary collection)
  tags = {
    ...
  }
 
  ebs_block_device {
    device_name  = local.dr_ec2_instances[each.key].ami_rbd_device_name         <-- (secondary collection)
    volume_type  = local.dr_ec2_instances[each.key].ami_rdb_volume_type         <-- (secondary collection)
    iops         = ...
    throughput   = ...
    snapshot_id  = data.aws_ebs_snapshot.dr_snap[each.key].id   <-- (base collection)
    delete_on_termination = true
  }
}
4

Have you tried kotowick’s deepmerge module?