Issue With random_shuffle - Item Not Found for New Items

Terraform
1

I’m having an issue where when I add new items to a list being used as the input for random_shuffle it doesn’t seem to recognize that the random_shuffle will change, and blows up if you try to reference the index of a newly added item.

For example:

Original Terraform

locals {
  things = [
    "CW1",
    "CW2",
    "CW3"
  ]
}

resource "random_shuffle" "random_things" {
  input = local.things
}

output "random_things" {
  value = random_shuffle.random_things.result
}

output "random_index_test" {
  value = index(random_shuffle.random_things.result, "CW3")
}

Original Plan

Terraform will perform the following actions:
  # random_shuffle.random_things will be created
  + resource "random_shuffle" "random_things" {
      + id     = (known after apply)
      + input  = [
          + "CW1",
          + "CW2",
          + "CW3",
        ]
      + result = (known after apply)
    }
Plan: 1 to add, 0 to change, 0 to destroy.
Changes to Outputs:
  + random_index_test = (known after apply)
  + random_things     = (known after apply)

Original Apply:

random_shuffle.random_things: Creating...
random_shuffle.random_things: Creation complete after 0s [id=-]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
random_index_test = 1
random_things = [
  "CW1",
  "CW3",
  "CW2",
]

But if I add another item to the original list of things being shuffled, and try to look up the index of that item in the shuffled list, it doesn’t recognize that the shuffled list should be changed and tries to lookup the item in the old shuffled list where the item doesn’t exist, and fails in the Plan job with an item not found error.

Example:

Updated Terraform: (added CW4 item)

locals {
  things = [
    "CW1",
    "CW2",
    "CW3",
    "CW4"
  ]
}

resource "random_shuffle" "random_things" {
  input = local.things
}

output "random_things" {
  value = random_shuffle.random_things.result
}

output "random_index_test" {
  value = index(random_shuffle.random_things.result, "CW4")
}

Updated Plan: (failure)

random_shuffle.random_things: Refreshing state... [id=-]
Error: Error in function call
  on main.tf line 21, in output "random_index_test":
  21:   value = index(random_shuffle.random_things.result, "CW4")
    |----------------
    | random_shuffle.random_things.result is list of string with 3 elements
Call to function "index" failed: item not found.

Seems like with the original/first plan, it should recognize that these things will be (known after apply) and not try to resolve them in the plan.

Any thoughts?

I’ve tried adding dependencies to the shuffle on the output, and messing with keepers on the shuffle, but no dice.

2

The above was just a simple example to reproduce the issue, but for more context what we’re actually trying to do is have a different random list of CWLG in each account/region, where the index of a CWLG in the random list determines the schedule for when the CWLG will be archived, since there can only be one export task at a time. And we didn’t want all accounts/regions to be trying to export the same CWLG at the exact time, since some are much larger than others (CloudTrail vs. SES for example) - we didn’t want all accounts doing all the big ones at the same time.

Right now as a workaround, we just rename the random_shuffle between something like random_things and rand_things whenever we add new items to the list, so that the shuffle will forcibly get re-created, but that’s kind of jankey.

3

If I just add the new item, without trying to reference the new item’s index in the shuffle, the plan will correctly recognize that the shuffle will be replaced and result not known until after the plan.

Terraform will perform the following actions:
  # random_shuffle.random_things must be replaced
-/+ resource "random_shuffle" "random_things" {
      ~ id     = "-" -> (known after apply)
      ~ input  = [ # forces replacement
            "CW1",
            "CW2",
            "CW3",
          + "CW4",
        ]
      ~ result = [
          - "CW1",
          - "CW3",
          - "CW2",
        ] -> (known after apply)
    }
Plan: 1 to add, 0 to change, 1 to destroy.
4

Which version of Terraform are you seeing this behaviour with? I can’t reproduce it in any version since 0.14.0, although it is present in 0.13.6. If you’re using an old version, can you try upgrading to the latest 1.0 patch release?

5

This seems like a behavior that would crop up under Terraform’s old model of implementing the “refresh” step as an entirely separate process than the subsequent “plan” step, which meant that Terraform would first need to evaluate all of the output value expressions against the refreshed state before proceeding to create the plan and learn that a new instance would be created.

Modern Terraform does refreshing and planning together in the same pass, and so it can deal with both before it tries to update the output values. I expect that’s why this doesn’t reproduce on modern versions of Terraform, and so I’d second @alisdair’s suggestion to try upgrading to the latest v1.0 release.

6

Thanks guys. We are currently using 0.13.7 I’ll retest when we eventually get to v1.