Create a resource of every resource created by for_each

Terraform
1

Hello,

I have a resource of aws_kinesis_stream that is created by using a for_each expression from a variable that contains a map.

variable "stream_envs" {
  default = {
    dev = "dev-stream"
    staging = "staging-stream"
    prod = "prod-stream"
  }
}

resource "aws_kinesis_stream" "cwl_kinesis_stream" {
  for_each = var.stream_envs
  name             = each.value
  shard_count      = 1
  retention_period = 24

  shard_level_metrics = [
    "IncomingBytes",
    "OutgoingBytes",
    "IncomingRecords",
    "OutgoingRecords",
    "WriteProvisionedThroughputExceeded",
    "ReadProvisionedThroughputExceeded",
    "IteratorAgeMilliseconds"
  ]

  tags = {
    Environment = each.key
    Team        = "Devops"
    managed_by  = "Terraform"
    Role        = each.value
  }
}

This portion looks to be working as expected. However, I want to create an aws_kinesis_firehose_delivery_stream that uses my Kinesis stream as a source for every environment. For example, dev kinesis stream correlates to dev firehose stream

resource "aws_kinesis_firehose_delivery_stream" "s3_delivery_stream" {
  for_each = var.stream_envs
  name        = format("%s-firehose_s3_destination", each.key)
  destination = "extended_s3"

  kinesis_source_configuration {
    kinesis_stream_arn = <kinesis stream>
    role_arn = "arn:aws:iam::my_role"
  }

  extended_s3_configuration {
    role_arn           = "arn:aws:iam::my_role"
    bucket_arn         = "arn:aws:s3:::my_bucket"
    buffer_size        = "5"
    buffer_interval    = "300"
    compression_format = "UNCOMPRESSED"

    processing_configuration {
      enabled = "true"

      processors {
        type = "Lambda"

        parameters {
          parameter_name  = "LambdaArn"
          parameter_value = "arn:aws:lambda::my_lambda:$LATEST"
        }
      }
    }
  }

  tags = {
    Environment = each.key
    Team        = "Devops"
    managed_by  = "Terraform"
    Role        = format("%s-firehose_s3_destination", each.key)
  }

  depends_on = [
    aws_kinesis_stream.cwl_kinesis_stream,
  ]
}

How can I achieve this?

Thanks

2

Hi @popopanda!

The most direct answer to your question with how you have your configuration written right now is that you can use aws_kinesis_stream.cwl_kinesis_stream itself as a map whose keys are the stream_envs keys:

  kinesis_stream_arn = aws_kinesis_stream.cwl_kinesis_stream[each.key].arn

With that said, because for_each works with map values and aws_kinesis_stream.cwl_kinesis_stream is itself a map value, a more direct way to express the intent “create one firehose delivery stream per kinesis stream” is to use the kinesis stream resource itself as the for_each value:

resource "aws_kinesis_firehose_delivery_stream" "s3_delivery_stream" {
  for_each = aws_kinesis_stream.cwl_kinesis_stream

  # ...
}

With this value of for_each, you can use each.value to access the attributes of the kinesis stream object:

  kinesis_stream_arn = each.value.arn

The keys in aws_kinesis_stream.cwl_kinesis_stream match the keys in var.stream_envs, so you can still use each.key to get the stream environment name as before. If you need access to the values of var.stream_envs inside the delivery stream resource then you can use each.value.name and thus access the name indirectly via the kinesis stream object.

Note that the depends_on containing aws_kinesis_stream.cwl_kinesis_stream is unnecessary with both of the above techniques because Terraform can infer that dependency automatically from the other references.

1 Like