LXD: lxc move to remote server forces terraform to recreate container?

I have two LXD (Ubuntu 24.04) servers in a remote relationship (non-cluster) and when I move a container from one host to the other terraform has no idea so it want’s to recreate the resource… I get it.

If I want to rename a resource I can do a state mv

$ terraform state mv lxd_instance.java lxd_instance.java1
Move "lxd_instance.java" to "lxd_instance.java1"
Successfully moved 1 object(s).

But if I want to just change a property like remote = null to remote = "srv2" then I need to recreate the resource

  # lxd_instance.mwsql must be replaced
-/+ resource "lxd_instance" "java1" {
      + interfaces       = (known after apply)
      + ipv4_address     = (known after apply)
      + ipv6_address     = (known after apply)
      + mac_address      = (known after apply)
        name             = "java1"
      ~ profiles         = [
          + "autostart",
            "plad",
          + "hddpool",
          + "ssdpool",
          + "static_ip_50",
        ]
      - remote           = "srv2" -> null # forces replacement
      ~ status           = "Stopped" -> (known after apply)
      ~ target           = null -> (known after apply)
        # (8 unchanged attributes hidden)
    }

Can I just go into the state file and update it? I did this once and it totally worked. Then I made some other changes and boom, now I’m being told the server has to be recreated

Terraform will perform the following actions:

  # lxd_instance.java1 must be replaced
-/+ resource "lxd_instance" "java1" {
      ~ interfaces       =  (known after apply)
      ~ ipv4_address     = "10.10.10.50" -> (known after apply)
      + ipv6_address     = (known after apply)
      ~ mac_address      = (known after apply)
      ~ name             = "java" -> "java1" # forces replacement
      ~ status           = "Running" -> (known after apply)
      ~ target           = null -> (known after apply)
        # (9 unchanged attributes hidden)
    }

What terraform command do I use to reliably let terraform know that the instance I just moved from one LXD host to another is the same instance as before just at a new location?

Renaming a resource in the state or the config has no bearing on the remote resource. The addresses use by Terraform are only so that it can map between the configuration and the state of each resource instance. So using moved blocks, or state mv and changing the config according is all done locally with no input from the provider or remote APIs.

The determination to replace a resource instance is not made by Terraform, but rather by the provider. The provider knows which API attributes can be updated in-place via the API, and which ones require replacing the entire resource, and will report during the plan when an update cannot be made in-place. The provider also informs Terraform which attribute in the change caused the replacement, which is notated in the plan output with # forces replacement.

Thank you for pointing out the responsibilities of terraform vs the provider.

I’m sorry I wasn’t clear, I am directly making the change at the provider level, in this case I’m moving an instance from srv1 to srv2. I’m then going into terraform and trying to inform (trick) it that the instance srv1:java is now srv2:java so it can know it doesn’t need to be recreated… I was not trying nor expecting terraform to do the instance migration.

It sounds like if I manually move an instance as stated then I just need to make the tf file match the state file and then terraform will be unaware anything “changed” since when it asks the provider “hey, I need to make an instance called srv2:java” and the provider says “it’s already there my guy, nothing to do”

If that is kinda the idea then when I need to make manual changes at the provider level and then backfill the info into terraform I backfill by manually ensuring the state and tf files match what the provider will report?

I don’t fully grasp what you mean by doing it “at the provider level”, if you mean you are changing the provider code itself, or even the provider version, the provider would be responsible for normalizing differences in the configuration.

You can rename and refactor configuration objects, but when you see something like

~ name = "java" -> "java1" # forces replacement

that is part of the resource configuration and the change must be taken into account by the provider. The provider knows that changing the name attribute remotely requires creating a new instance entirely. In order to prevent that change, the configuration must match what the provider sees from the APIs.

If you have somehow changed the real-world name to java1 and the provider is not reporting that, and indicating a change when you update the config, that would be a bug in the provider. In that case perhaps manually updating the state to match the real-world is an acceptable workaround.

Sorry for the confusion about “provider level”… what I’m doing is one of two things:

  1. renaming a container instance in the LXD host
  2. moving a container instance from one LXD host to another remote LXD host
  3. refactoring my terraform files to make them more maintainable

RE: #1
When the container named java changes to java1 using the LXC command lxc move java java1 terraform has nothing to do with this and the change isn’t know to it.
When I subsequently run terraform plan then the magic of terraform talks to the magic of LXD and they determine a destroy/create has to take place in order to make a java1 container exist and to get rid of java since thats what terraform really “knows” about.
As the human that went outside of terraform to make a change I know what really needs to happen is “java is now java1 so just rename it in the tf files”
To manually make terraform aware of my human meddling I can do the following:

  • use the command terraform state mv lxd_instance.java lxd_instance.java1
  • manually edit the config lxd_instance.java1 changing name = "java1"
  • manually edit the state file for lxd_instance.java1 changing name = "java1"

I was asking

  1. is there a command to make terraform aware of my original LXC changes instead of doing the manual steps?
  2. is there a terraform command that would be the equivalent of lxc move java java1 which doesn’t destroy and recreate the instance?

RE: #2
Similarly to #1 but the command is lxc move java1 srv2:java1 where srv2: signifies move the instance to a remote LXD server.
Terraform is somewhat aware of remotes as a property of lxd_instance and such. But if I change the tf config for the instance to remote = "srv2" then terraform wants to destroy/create again. I again have to trick it by manually migrating it to the remote then backfilling the terraform files like in #1

RE: #3
When I refactor my terraform code I use terraform state mv lxd_instance.instances[0] lxd_instance.java1 for example… this seems like kinda what I want to do for the additional properties with the configuration of an instance. like:
terraform state mv lxd_instance.java1.remote lxd_instance.java1.remote="srv2"
(I have no idea if dot notation is valid in terraform, but that kinda feels like it should be)
or maybe
terraform state add lxd_instance.java1.remote="srv2" (add/edit/update/etc)

Summary
Fundamentally I know that terraform should be the thing that makes the changes in the environment, but if there is no “move without destroy/create” command through terraform and there is natively in LXC then I just wanna backfill that change in terraform as easily as possible so when the next terraform apply is run it doesn’t try and undo the manual changes.
I have the workaround manual steps that are painfully working for now, just didn’t know if there was a better way.
While typing this I’m sure if I had the container instance using NFS or other remote storage and configured the destroy/create to simply mount the storage again then this would be another way to solve the host rebalancing, but thats not possible in this environment at the moment.

Thank you for your time and energy… you rock!

I think there are some misunderstands here which are causing confusion. The name of Terraform configuration objects, i.e. what you use in the address of each resource, is completely unrelated to what the provider sees. You could rename every resource, or no resources, and none of that would produce any changes in the remote infrastructure.

The state commands can’t modify the state of the individual resource instances. Terraform can only change the address which it uses to identify the resource internally. Using moved blocks in the config makes this more clear, and gives to immediate feedback within the plan to make sure the moves align with the configuration, but again, that has no bearing on the provider, the remote instances, or their individual attributes.

So any changes you make externally will correspond to some attributes within the resource instances. How that’s handled is entirely up to the provider implementation and the capabilities of the remote api.

  • If the provider can no longer locate the instance because some primary identifier has changed, then Terraform will report the instance as missing and it will need to be imported.
  • If the instance is located, but the configuration no longer matches the remote instance, then Terraform will attempt to restore what is in the configuration (the configuration always represents your desired final state). You then must update the configuration to match what has changed externally.

There is no command to alter the resource instance attributes. That data is managed by the provider, and any changes there will be made during the various calls sent to the provider during the plan and apply phases. If there are changes which are not being made for some reason, that is a problem within the provider. The only situation I can think of here would be: the remote instance changed, the configuration was updated to match the changes made externally, but the provider is not reading back the updated remote state for some reason. This could result in unexpected changes during plan which are not resolvable because the provider is not cooperating.

The only options if the provider is preventing the change would be to either remove and import the resource, or alter the state file directly. This is the least likely situation however, so review carefully the steps which were taken before throwing out the existing state.

Thank you for your patience and help! I’m sorry I am less familiar with the Terraform vernacular adding to the difficulty in communication.

Thank you for pointing out that terraform is enforcing the desired state.

This sounds like the thing I need is an anti pattern and if I’m able to find a hacky workaround then I should regard it as that, a hack.

If the desired state and the remote state match then terraform will take no action. Simple.

If there is a delta between the desired state and the remote state then terraform makes choices of how to get the desired state implemented on the remote provider. The remote provider simply reports state and executes the API commands terraform requests.

Does that about sum it up?

Feel free to set me straight if you want. Seems my action item is “make the remote and desired states match” either in a hacky manual way or letting terraform do what it was designed to do.