Making an AWS S3 bucket with multiple event notifications for multiple SQS queues?

Terraform Providers AWS
1

Context:
I have an AWS environment where I currently have multiple buckets, each with their own set of event notifications.

Each event notification maps a certain prefix filter to a unique SQS queue.

I wrote some TF to automate this and I pass in a env_name.tfvars file (which contains a map(map(object({ prefix = string }))) that lists all the queues that need to be made as well as the prefixes that need to be associated with each of them) and I’m able to make Buckets and SQS queues with no problem via for_each.

Problem:
But when the s3 event notifications are being created, I see that they create successfully, yet when I look at the bucket properties in the AWS console, I see that only one event notification has been created.

This is “expected behavior” for some reason, and I’m confused as to why? It’s very much possible to create multiple event notifications for a given bucket manually in the AWS Console, why can’t terraform do it?

I don’t understand why “Declaring multiple aws_s3_bucket_notification resources to the same S3 Bucket will cause a perpetual difference in configuration.”

Given my use case (multiple buckets, each with own event notifications pointing to multiple different SQS queues based on supplied prefix), how can I get around this seemingly artificial limitation?

2

Hi @pritster5,

I must admit to not being very familiar with this part of S3, but it seems like aws_s3_bucket_notification corresponds with the underlying S3 API action s3:PutBucketNotificationConfiguration.

According to the API docs, it seems like each S3 bucket has only one “notification configuration” object, and so this API action always overwrites any existing notification configuration when called.

However, a single “notification configuration” object can have zero or more queues, so you can write a single notification configuration resource with multiple queue blocks if you need to send notifications to multiple locations:

resource "aws_s3_bucket_notification" "example" {
  queue {
    id        = "example-1"
    queue_arn = aws_sqs_queue.example1.arn
    # ...
  }
  queue {
    id        = "example-2"
    queue_arn = aws_sqs_queue.example2.arn
    # ...
  }
}

To generate queue blocks dynamically based on an expression, you could use a dynamic block instead, like this:

resource "aws_s3_bucket_notification" "example" {
  dynamic "queue" {
    for_each = local.example
    content {
      id        = queue.value.id
      queue_arn = queue.value.arn
      # ...
    }
  }
}

A dynamic block is a similar idea to resource for_each, but it’s more like a “macro” that generates normal configuration blocks as far as the provider is concerned, whereas resource-level for_each actually declares multiple separate resource instances that are all independent of one another from the provider’s standpoint.

For your purposes, you should be able to populate for_each for the dynamic "queue" block in the same way you would’ve populated a resource-level for_each, and then use queue.value to refer to the current element’s value, instead of each.value.

1 Like
3

This is precisely what I needed, thanks so much! I’m pretty new to TF so I wasn’t aware of Dynamic. This makes things a lot easier, though I will have to rework my module composition a bit.

Thanks again!