For_each support sequential operation?

Terraform
1 
module "app" {
     for_each        = var.account_info
     source          = "./db-operations"
     dbrole          = each.key
     database_name   = var.database_name
     secret_path     = each.value
     db_password_key = var.db_password_key
     db_user_key     = var.db_user_key
   }

I have the above code and it throws the error

Error: could not execute revoke query: pq: tuple concurrently updated

The reason for that is because the module tries to update the database (grant/revoke, and role creation). Terraform is doing it concurrently. So if the size of var.account_info is 1 there won’t be any error, if the size is 2 I will have to terraform plan/apply twice.

Is it possible for me to specify terraform to run for_each sequentially?

1 Like
2

Hi @keithlin.732,

Terraform Core considers all instances of a single resource to be independent from one another (they cannot refer to each other, and therefore cannot have interdependencies) and so often the plan and apply operations for these will happen concurrently, from Terraform Core’s perspective.

When an underlying system has additional restrictions on what actions can happen concurrently, or on how quickly and often certain operations may happen, in Terraform architecture it’s conventionally the provider’s responsibility to use a strategy like a mutex or a semaphore to handle the concurrent requests from Terraform as sequential requests to the remote system. Because the Terraform language focuses on describing desired state rather than the the specific actions to achieve that state, there is no way to express that within the Terraform language itself.

You didn’t mention exactly which provider you are using here but I recognize this as a Postgres error and so I suppose you are using the community provider cyrilgdn/postgresql. If the maintainer(s) would be willing, the most ideal solution would be for the provider to handle this error itself, either by using a synchronization primitive like I described above or by recognizing this particular error and politely retrying the operation for some reasonable period of time before totally failing the operation.

In the meantime though, a possible workaround is to instruct Terraform Core to perform all operations sequentially. You can use terraform apply -parallelism=1 to tell Terraform to set its own internal semaphore to perform no more than one operation at a time. That option is a global one intended to make Terraform use fewer resources on constrained systems, but by setting it to 1 you effectively prevent Terraform from doing any concurrent operations at all, which should avoid the problem you encountered but at the expense of preventing any other concurrency for any other resources too.

1 Like
3

Hi @apparentlymart,

I know this is now 3 years old, but I just stumbled upon a similar case, where I have several instances in a cluster, and they’re deployed using a for_each because that’s a simpler way than having the same module repeated 20 times in code.

However, when I perform an update of that cluster, I’d like these instances to be updated in order, one by one, so that the impact on the running cluster would be minimal.

I was wondering how hard would it be to implement something like for_each_sequence which would accept an ordered list of keys specified in for_each, telling terraform in what sequence the instances of the resource / module need to be executed. It could look like this:

module "vm" {
  for_each = toset(var.nodes)

  # One by one, in alphabetical order.
  for_each_sequence = sort(var.nodes)
  ...
}

or:

module "vm" {
  for_each = toset(vm.names)

  # I don't care about the order, I just want them one-by one.
  for_each_sequence = vm.names
  ...
}
1 Like
4

NVM, Just found Execution order within `for_each` loop · Issue #30841 · hashicorp/terraform · GitHub, moving the discussion there.

1 Like