Preserve order with dynamic block

Terraform
1

Hello folks,
I’m trying to solve an issue with preserving creation order in a dynamic block.
the code accept a map with the queue name and the bucket prefix:

  sqs_queues = {
    "errors-test-queue"    = "errors"
    "files-test-queue"     = "files"
    "job-test-queue"       = "jobs"
  }

I’ve the following tf code:

variable "sqs_queues" {
  description = "SQS queue function map."
  type        = map(string)
  default     = {}
}

data "aws_sqs_queue" "sqs_queues" {
  for_each = var.sqs_queues
  name     = each.key
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  count  = length(data.aws_sqs_queue.sqs_queues) != 0 ? 1 : 0
  bucket = aws_s3_bucket.bucket.id

  dynamic "queue" {
    for_each = data.aws_sqs_queue.sqs_queues
    content {
      queue_arn     = queue.value.arn
      events        = ["s3:ObjectCreated:*"]
      filter_prefix = var.sqs_queues[queue.value.name]
    }
  }
}

Every time i add a new queue/prefix pair the dynamic resources created in the bucket_notification shift and terraform wants to recreate them causing downtime.
i’ve tried with no success to convert the input to a list or the data obj to a list but it seems not working.

2

You say dynamic resources, but actually these are dynamic blocks within a single resource.

There’s no way you can prevent items shifting, because the underlying dynamic block is of list type:

I’m not sure that actually matters, though, because regardless how you change the dynamic blocks, the whole bucket notification configuration is being written back to AWS as a single API call:

I’m not familiar enough with AWS to be able to speculate whether this applies atomically or not, but either way, Terraform is just making the API call it needs to … I don’t think there’s anything you, or Terraform, could do differently here.

3

thank you for looking into this maxb,
we had downtime we saw notifications being sent to the wrong queues, we had to plan and re-apply tf and that fixed it.
a bucket can have a single notification obj, so this is the only way for me to deal with adding multiple notification settings to the same bucket.

4

That is very puzzling… I don’t suppose you have the plan and apply logs, both for the first time, which resulted in things not working properly, and for the second time to fix it?

5

Hey max, I work with @ltagliamonte. We have another service which polls the queues and that service had to be reapplied when it started forwarding the wrong messages

6

I haven’t been yet able to find a solution to this problem.
we can’t use multiple aws_s3_bucket_notification resources because a bucket can have only one defined (if i create multiple resources they override each other).

this is blocking us to provision new queues.

7

No-one answered my question about logs, which to me feels like the necessary next step in understanding this.

The question is kind of interesting, but that’s not enough for me as a private individual to want to spend money on billable AWS operations just to try to reproduce it.

8

hello maxb, thank you for the reply.
i created a test env to replicate and this is the tf output when i plan after trying to add a new queue:

  # module.iot-s3-uploader-lt-test.aws_s3_bucket_notification.bucket_notification[0] will be updated in-place
  ~ resource "aws_s3_bucket_notification" "bucket_notification" {
        id          = "lt-test-bucket"
        # (2 unchanged attributes hidden)

      ~ queue {
          ~ filter_prefix = "hdsfiles" -> "sessclip"
            id            = "tf-s3-queue-20230225202032611400000002"
          ~ queue_arn     = "arn:aws:sqs:us-west-1:<REDACTED>:003-lt-test-queue" -> "arn:aws:sqs:us-west-1:<REDACTED>:002-lt-test-queue"
            # (1 unchanged attribute hidden)
        }
      ~ queue {
          ~ filter_prefix = "jobresults" -> "hdsfiles"
            id            = "tf-s3-queue-20230225202032611400000003"
          ~ queue_arn     = "arn:aws:sqs:us-west-1:<REDACTED>:004-lt-test-queue" -> "arn:aws:sqs:us-west-1:<REDACTED>:003-lt-test-queue"
            # (1 unchanged attribute hidden)
        }
      + queue {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "jobresults"
          + queue_arn     = "arn:aws:sqs:us-west-1:<REDACTED>:004-lt-test-queue"
        }
        # (1 unchanged block hidden)
    }

initially the module got initialised with:

  sqs_queues = {
    "001-lt-test-queue" = "errorreport"
    "003-lt-test-queue" = "hdsfiles"
    "004-lt-test-queue" = "jobresults"
  }

i later added: "002-lt-test-queue" = "sessclip" to the sqs_queues map.
no matter how i name the queues the result is always a shit of the notifications.

9

I proposed that it would be helpful to see

But you’ve only provided the plan for the first time. I have no further insight to give based on this information.

10

Hello maxb thank you for the interest in the issue, following is the whole repro of the issue.
I’ve created 3 queues:

  • lt-errorreport
  • lt-jobresults
  • lt-clips

and i’ve created a bucket lt-tf-test-bucket and i’m going to use the following tf code to create bucket notifications to the first 2 queues i previously created:

provider "aws" {
  profile = "<REDACTED>"
  region  = "us-west-1"
}

locals {
  sqs_queues = {
    "lt-errorreport" = "errors"
    "lt-jobresults"  = "results"
  }
}

data "aws_sqs_queue" "sqs_queues" {
  for_each = local.sqs_queues
  name     = each.key
}

resource "aws_sqs_queue_policy" "allow_bucket_sqs" {
  for_each = data.aws_sqs_queue.sqs_queues

  queue_url = each.value.url

  policy = <<POLICY
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "sqs:SendMessage",
            "Resource": "${each.value.arn}",
            "Condition": {
              "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::lt-tf-test-bucket" }
            }
          }
        ]
      }
  POLICY
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  count  = length(data.aws_sqs_queue.sqs_queues) != 0 ? 1 : 0
  bucket = "lt-tf-test-bucket"

  dynamic "queue" {
    for_each = data.aws_sqs_queue.sqs_queues
    content {
      queue_arn     = queue.value.arn
      events        = ["s3:ObjectCreated:*"]
      filter_prefix = local.sqs_queues[queue.value.name]
    }
  }
}

here is the initial plan that i’m going to apply:

terraform plan -out tfplan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket_notification.bucket_notification[0] will be created
  + resource "aws_s3_bucket_notification" "bucket_notification" {
      + bucket      = "lt-tf-test-bucket"
      + eventbridge = false
      + id          = (known after apply)

      + queue {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "errors"
          + id            = (known after apply)
          + queue_arn     = "arn:aws:sqs:us-west-1:<REDACTED>:lt-errorreport"
        }
      + queue {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "results"
          + id            = (known after apply)
          + queue_arn     = "arn:aws:sqs:us-west-1:<REDACTED>:lt-jobresults"
        }
    }

  # aws_sqs_queue_policy.allow_bucket_sqs["lt-errorreport"] will be created
  + resource "aws_sqs_queue_policy" "allow_bucket_sqs" {
      + id        = (known after apply)
      + policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sqs:SendMessage"
                      + Condition = {
                          + ArnEquals = {
                              + aws:SourceArn = "arn:aws:s3:::lt-tf-test-bucket"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = "*"
                      + Resource  = "arn:aws:sqs:us-west-1:<REDACTED>:lt-errorreport"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + queue_url = "https://sqs.us-west-1.amazonaws.com/<REDACTED>/lt-errorreport"
    }

  # aws_sqs_queue_policy.allow_bucket_sqs["lt-jobresults"] will be created
  + resource "aws_sqs_queue_policy" "allow_bucket_sqs" {
      + id        = (known after apply)
      + policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sqs:SendMessage"
                      + Condition = {
                          + ArnEquals = {
                              + aws:SourceArn = "arn:aws:s3:::lt-tf-test-bucket"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = "*"
                      + Resource  = "arn:aws:sqs:us-west-1:<REDACTED>:lt-jobresults"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + queue_url = "https://sqs.us-west-1.amazonaws.com/<REDACTED>/lt-jobresults"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"
x:queues ltagliamonte$ terraform apply "tfplan"
aws_sqs_queue_policy.allow_bucket_sqs["lt-errorreport"]: Creating...
aws_s3_bucket_notification.bucket_notification[0]: Creating...
aws_sqs_queue_policy.allow_bucket_sqs["lt-jobresults"]: Creating...
aws_s3_bucket_notification.bucket_notification[0]: Creation complete after 1s [id=lt-tf-test-bucket]
aws_sqs_queue_policy.allow_bucket_sqs["lt-jobresults"]: Still creating... [10s elapsed]
aws_sqs_queue_policy.allow_bucket_sqs["lt-errorreport"]: Still creating... [10s elapsed]
aws_sqs_queue_policy.allow_bucket_sqs["lt-errorreport"]: Still creating... [20s elapsed]
aws_sqs_queue_policy.allow_bucket_sqs["lt-jobresults"]: Still creating... [20s elapsed]
aws_sqs_queue_policy.allow_bucket_sqs["lt-errorreport"]: Creation complete after 26s [id=https://sqs.us-west-1.amazonaws.com/<REDACTED>/lt-errorreport]
aws_sqs_queue_policy.allow_bucket_sqs["lt-jobresults"]: Creation complete after 26s [id=https://sqs.us-west-1.amazonaws.com/<REDACTED>/lt-jobresults]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

after applied i’ve changed the local variable to be:

locals {
  sqs_queues = {
    "lt-errorreport" = "errors"
    "lt-jobresults"  = "results"
    "lt-clips"       = "clips"
  }
}

here is the plan i get now, with the shifted resources:

terraform plan -out tfplan
aws_sqs_queue_policy.allow_bucket_sqs["lt-jobresults"]: Refreshing state... [id=https://sqs.us-west-1.amazonaws.com/082346306812/lt-jobresults]
aws_sqs_queue_policy.allow_bucket_sqs["lt-errorreport"]: Refreshing state... [id=https://sqs.us-west-1.amazonaws.com/082346306812/lt-errorreport]
aws_s3_bucket_notification.bucket_notification[0]: Refreshing state... [id=lt-tf-test-bucket]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket_notification.bucket_notification[0] will be updated in-place
  ~ resource "aws_s3_bucket_notification" "bucket_notification" {
        id          = "lt-tf-test-bucket"
        # (2 unchanged attributes hidden)

      ~ queue {
          ~ filter_prefix = "errors" -> "clips"
            id            = "tf-s3-queue-20230226233626104600000001"
          ~ queue_arn     = "arn:aws:sqs:us-west-1:082346306812:lt-errorreport" -> "arn:aws:sqs:us-west-1:082346306812:lt-clips"
            # (1 unchanged attribute hidden)
        }
      ~ queue {
          ~ filter_prefix = "results" -> "errors"
            id            = "tf-s3-queue-20230226233626104600000002"
          ~ queue_arn     = "arn:aws:sqs:us-west-1:082346306812:lt-jobresults" -> "arn:aws:sqs:us-west-1:082346306812:lt-errorreport"
            # (1 unchanged attribute hidden)
        }
      + queue {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "results"
          + queue_arn     = "arn:aws:sqs:us-west-1:082346306812:lt-jobresults"
        }
    }

  # aws_sqs_queue_policy.allow_bucket_sqs["lt-clips"] will be created
  + resource "aws_sqs_queue_policy" "allow_bucket_sqs" {
      + id        = (known after apply)
      + policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sqs:SendMessage"
                      + Condition = {
                          + ArnEquals = {
                              + aws:SourceArn = "arn:aws:s3:::lt-tf-test-bucket"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = "*"
                      + Resource  = "arn:aws:sqs:us-west-1:082346306812:lt-clips"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + queue_url = "https://sqs.us-west-1.amazonaws.com/082346306812/lt-clips"
    }

Plan: 1 to add, 1 to change, 0 to destroy.