Terraform deletion and update order issue

[henryrecker]

We are running into an issue where we are forced to set lifecycle.create_before_destroy = true or run a two-step apply to avoid terraform running API calls in the wrong order, and I want to check if this is expected.

We start with a configuration like this:

resource "example_dependency" "dependency_a" {
  name = "dependency_a"
}

resource "example_dependency" "dependency_b" {
  name = "dependency_b"
}

resource "example_resource" "main" {
  name = "main_resource"

  children = [
    example_dependency.dependency_a.id,
  ]
}

We want to run an apply to both remove the dependency_a resource and update the main resource to point to dependency_b instead. New config:

# dependency_a removed from config

resource "example_dependency" "dependency_b" {
  name = "dependency_b"
}

resource "example_resource" "main" {
  name = "main_resource"

  children = [
    # changed
    example_dependency.dependency_b.id,
  ]
}

The problem is that terraform tries to delete dependency_a before it tries to update main. The API call then fails because main is still dependent on that object.

Our workaround is to use lifecycle.create_before_destroy, or run the change to main first as a separate apply.

Is this expected, or should terraform be ordering these correctly?

[apparentlymart]

What I understood from the question is that example_dependency.dependency_b is planned to be replaced, while example_resource.main is planned to be updated. Terraform then breaks the “replace” into separate “delete” and “create” steps, and so you’re observing the following order:

1. Delete the old example_dependency.dependency_b
2. Create the new example_dependency.dependency_b
3. Update example_resource.main to refer to the new example_dependency.dependency_b

Terraform cannot update example_resource.main until it knows the new value of example_dependency.dependency_b, and in the “delete then create” replace ordering the delete must happen before the create, and so this seems like the only workable order of these operations.

You mentioned that you were able to get the behavior you wanted by using create_before_destroy, and so I assume (but haven’t checked) that’s causing Terraform to use the following order instead:

1. Create the new example_dependency.dependency_b
2. Update example_resource.main to refer to the new example_dependency.dependency_b
3. Delete the old example_dependency.dependency_b

In this case the new value of example_dependency.dependency_b.id can be known before the old example_dependency.dependency_b has been destroyed.

Another different way to do it would be to configure example_resource.main with replace_triggered_by = [example_dependency.dependency_b.id], which should then force example_resource.main to also get replaced in this case, leading to the following order (as long as you aren’t also using create_before_destroy):

1. Delete the old example_resource.main
2. Delete the old example_dependency.dependency_b
3. Create the new example_dependency.dependency_b
4. Create the new example_resource.main, referring to the new example_dependency.dependency_b

or, if you use both replace_triggered_by and create_before_destroy together:

1. Create the new example_dependency.dependency_b
2. Create the new example_resource.main, referring to the new example_dependency.dependency_b
3. Delete the old example_resource.main
4. Delete the old example_resource.main

I think the crux of the problem is that the object associated with example_resource.main cannot exist without the object associated with example_dependency.dependency_b, and so updating example_resource.main in-place isn’t actually a viable plan in this case but Terraform does not have enough information to conclude that itself. Using replace_triggered_by is how you can tell Terraform about the stronger dependency relationship between these objects to make it plan a workable order of operations.

Allowing providers to give Terraform more information to automatically resolve this situation was one of the motivations for my proposal in issue #22094, but with the currently-defined provider protocol there just isn’t any way for Terraform to infer this extra requirement automatically.

---

(When I say “stronger dependency” above I mean that a normal dependency in Terraform just means “actions for this must happen before actions for that”, but it doesn’t directly affect which actions are planned. replace_triggered_by expresses a different kind of dependency: “this becomes invalid if that is deleted”, or as I called it in that proposal on GitHub: “containment”.)

[jbardin]

To say the same thing in a slightly different way – yes create_before_destroy isn’t just a workaround, it’s the expected way to handle this type of registration pattern (the option could be called create_and_or_update_dependencies_before_destroy).

There are many resource types which require updates before their dependencies can be destroyed, and it’s usually up to the provider to document such cases, since as apparentlymart pointed out the protocol does not have a method for tracking these relationships.

[henryrecker]

Thanks for the replies, that all makes sense.

To clarify in my example there is no planned replacement for dependency_b, the only planned changes are to destroy dependency_a and to update main. But regardless it sounds like this is expected and the provider should document this requirement.

[apparentlymart]

Hmm yes sorry… now I re-read I realized I mixed up your dependency_a and dependency_b in my head while I was thinking about it. A common hazard of working with contrived placeholder names that all look kinda similar on the page.

What I wrote out is still roughly right but of course the destroy actions are for dependency_a rather than dependency_b, because that one still exists in the configuration.