Only create resources that don't already exist

Terraform
1

Hello - I’m working on my first terraform project and am hitting a snag. I’d appreciate any advice or guidance folks could offer!
Thanks in Advance!

Background: Goal and Problem

Goal:
I am looking for a way to create a terraform resource only when a real-world resource of the same name doesn’t already exist.

My basic setup includes:

terraform version
Terraform v1.5.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/http v3.4.0
  • a data source to get me a list of extant items (Unfortunately the provider I’m working with doesn’t have a data source I can check, so I’m using the http provider’s data source to return information about the items from an API endpoint)
  • a corresponding local that creates a list of item names from the response body
  • a resource with the for_each meta-argument uses setsubtract to compute a set of items to create by checking for their existence in the local list

An initial terraform apply works as expected to correctly identify existing item names, and create the resource if none exist with the same name.

Problem:
A subsequent terraform apply will delete the just-created resource

This appears to be happening due to subsequent terraform plan and terraform apply runs not refreshing the content of the data source (even when using refresh-only). Interestingly, running terraform refresh does update the content of the data source to include the just-created item name. This appears to cause my for_each meta-argument to produce a set that never includes the just-created item name, which triggers that resource’s destruction.

The common workaround of just import these existing items into terraform doesn’t meet my project’s requirement for terraform-managed resources to coexist with resources created outside of terraform (without taking over their management, which introduces the risk of their accidental destruction).

What I’ve tried

data.tf

data "http" "rules" {
  url = "${local.baseurl}/api/v1/rules"
}

locals {
  rules = toset([for i in jsondecode(data.http.rules.response_body).data : i.name])
}

main.tf

# current config
resource "provider_rule" "my_rule" {
  for_each = setsubtract(["My Rule"], local.rules)
  name = "My Rule"
}

# also tried
resource "provider_rule" "my_rule" {
  for_each = contains(local.rules, "My Rule") ? toset([]) : toset(["rule"])
  name = "My Rule"
}

outputs.tf

output "existing_rules" {
  value = local.rules
}

terraform output

existing_rules = toset([
  "My Rule",
  "foo bar",
])

subsequent terraform plan

# module.provider.rule.my_rule["My Rule"] will be destroyed
  # (because key ["My Rule"] is not in for_each map)
  - resource "rule" "my_rule" {
      - enabled          = true -> null
      - id               = "0000000000014A51" -> null
      - name             = "My Rule" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.
2

Hi @jeff-d,

What you are trying to achieve here is impossible, because your configuration is self-contradictory.

You have effectively told Terraform “this object should exist if it doesn’t exist”, which is impossible. Because Terraform works by making incremental changes to the real system, Terraform responds to this contradiction by just never converging, because the actions required to make the actual state match the desired state effectively change the desired state as a side-effect.

Terraform is a desired state system, and so it assumes you have a specific idea of how the system ought to be. That includes you making a decision about whether this configuration is either responsible for managing this object, or if the configuration is just using an object managed elsewhere.

If you are starting to use Terraform to manage a pre-existing system where objects already exist, and you now want Terraform to take over management of those objects, the appropriate path would be to do what Terraform calls “import”, which means to take an existing object in the remote system and pretend that Terraform originally created it for all future operations.

In that case, the typical process for adopting Terraform would be:

  • Write a Terraform configuration with resource blocks that describe the objects that already exist.

  • Use the terraform import subcommand to tell Terraform to bind an existing remote object to each resource instance address you’ve just declared in your configuration.

  • Run terraform plan to see what Terraform proposes.

    If you’ve successfully described the existing objects in your configuration, Terraform will announce that no changes are required.

    If Terraform does propose some changes, you can review them and decide either to change the configuration to better match the existing object (so that no change is required), or apply the change to modify the existing objects to match what you wrote in the configuration.

data blocks are only for situations where a configuration permanently depends on an object that’s not managed by this Terraform configuration. It describes a dependency on an external object, as opposed to an intent to manage that object with Terraform.

3

If terraform is “desired state” and the system is already in the desired state then why is this an error and a contradiction?

The truth of the problem is that Terraform is much more than a “desired state” tool, because it also attempts to manage the state transitions of the resources and thus you cannot simply describe the desired end state but you must as well concern yourself with whether or not the state is to be created, updated and destroyed as well.

4

Terraform also considers whether or not it “owns” the object as part of the state, so if a Terraform configuration declares that something is to be managed and that object already existed but yet isn’t known to be owned by the current configuration then the desired state and current state do not match.

The terraform import command is an imperative-shaped way to tell Terraform to pretend it had created something and therefore owns it, which is one way to create convergence between the desired and actual states when an object was originally managed outside of Terraform but will henceforth be “owned by” Terraform.

You are right that it’s a gap in Terraform’s current model that it can’t know (without operator intervention) whether an existing object is intended to be owned by the current Terraform configuration or not, and so it conservatively assumes that the answer is “not” to try to avoid damaging an object that another system is relying on. That gap results from the fact that, unlike most information Terraform tracks, that information lives exclusively in the Terraform state and has no backing in the remote system. Therefore it’s necessary to intervene manually using either terraform import or import blocks to tell Terraform what your intention is.

You may prefer to describe the behavior of Terraform in different conceptual terms than I did, but nonetheless the original configuration we were discussing effectively uses the actual state to decide the desired state, and therefore each change to the actual state modifies the desired state and therefore it cannot ever converge.