How to change current terraform structure?

Terraform
#1

I am currently provisioning our infrastructure with terraform and have the following structure

- modules
- providers.tf
- rds.tf 
and so on

Initially idea was to have identical production and staging environment. I am using different branches in git to represent the production and staging environment and using Terraform cloud to store the state. Currently, there are some resources that need to be different, Some resources need to be present in prod but not in staging.

I am looking to change the structure as

- modules
- common 
- staging 
- production

How to go about the following migrations. Any documentations or ideas will be helpful

#2

Hi @rohit,

Since you are using Terraform Cloud I guess you probably currently have two workspaces that both refer to the same directory containing providers.tf but refer to different branches. Because they are separate workspaces they each have a separate state.

The first thing to change then would be to reconfigure the Terraform Cloud workspaces to refer to the same branch but to different working directories, so that the staging workspace refers to that staging directory while the production workspace refers to the production directory.

The next thing to do would be to construct the content of the staging and production directories so that Terraform has enough information to understand how to reorganize the existing objects in the state to be at new addresses inside the common module.

Terraform’s refactoring features can help with that as long as you are using Terraform v1.1 or later. For example, I expect your new staging root module will now consist of a single module block calling the common module, which you can then annotate with a moved block for each of the resources that were formerly in the root module:

module "common" {
  source = "../common"

  # ...
}

moved {
  from = aws_rds_cluster.example
  to   = module.common.aws_rds_cluster.example
}

That example moved block tells Terraform that if it finds in the state a resource instance called aws_rds_cluster.example in the root module, it should pretend that it had originally been declared with the same name inside the common module instead. It will then do all of the normal planning steps in terms of that rewritten address.

If you move all of the configuration that was originally in your shared root module into common then you’d need to add a moved block like the one above for each of the resources that root module was originally declaring, marking for each one that its configuration now belongs to module.common.

If you fully specify all of the moved blocks in this way, you should find that Terraform will produce a plan that proposes to do nothing other than update the state to move these existing objects to their new addresses.

I’d suggest testing this out with some new temporary workspaces first just to get familiar with the process. First set them up to match your current design where both refer to the same configuration directory and apply the change to create all of the objects, and then make the changes I described above to see how Terraform handles the refactoring statements. Once you’ve seen how it works and you feel confident about how to proceed, you can then make the same changes in your real configuration with your real workspaces.

#3

Thanks, This was helpful.
You are correct, I am using different workspaces and both the environment have separate states.
I was also looking into moved, The problem here is that there are 200+ resources here. I might have to look into some kind of script that can automate adding the moved blocks.

#4

In case it’s helpful to make the code generation easier, note that it’s possible to write moved blocks in the JSON variant of the Terraform language too, and so it might be easier to generate JSON like the following instead of generating the native syntax, if you are working in a language that already has a JSON encoding library:

{
  "moved": [
    {
      "from": "aws_rds_cluster.example",
      "to": "module.common.aws_rds_cluster.example"
    },
    {
      "from": "aws_instance.example",
      "to": "module.common.aws_instance.example"
    }
  ]
}

You might also find it helpful to use terraform show -json before you make any other changes in order to capture a JSON representation of the current state, through which your script can find all of the existing resource instances to use as the manifest of all existing objects that would need to be moved.