Request for feedback: Config-driven refactoring

Hi all,

My name is Korinne, and I’m a Product Manager for Terraform :wave:

We’re currently working on a project that will allow users to more easily refactor their Terraform modules and configurations, set to be generally available in Terraform v1.1. From a high-level, the goal is to use moved statements to do things like:

The above actions are declared within a module’s own source code, which allows Terraform to automatically apply the changes as part of creating a plan. This feature is currently included in v1.1.0-alpha20211006, and we’d be really grateful to get feedback from the Terraform community. You can learn more about this new feature (along with additional use cases) in its draft documentation.


Why this is important

Up until now, the primary tool for refactoring configuration has been terraform state mv, used in conjunction with corresponding changes to the configuration. While this can accomplish simple refactoring operations, module authors and Terraform Cloud practitioners had no way to coordinate these changes themselves. Here’s a quick summary of the problems our different users faced with refactoring:

  • Terraform CLI (open-source) users: Refactoring had to be done manually, by constructing commands to rename the state objects, and coordinate those changes with the configuration. This was often error-prone, with little feedback about how to resolve those errors.

  • Terraform Cloud users: General refactoring within Terraform Cloud workflows was not really possible, since the only interface is to create a plan based on changes to the configuration. In order to do any refactoring, users would need to temporarily switch to using local state with the CLI, and make the changes out-of-band from the normal workflow. This also assumes the user has admin access on the workspace, as they’ll need access to the state file.

  • Module authors: Directly refactoring modules was not possible, since the names of existing state objects were not within their control. In order to refactor a module, a new major version would be released to indicate a breaking change, and a guide to manually upgrade the state would be supplied to the end-user.

With this new mechanism, users can plan, preview, and validate multiple refactoring actions simultaneously. This allows for:

  • Easier and safer refactoring: Common tasks which may have involved many individual commands, like moving individual resources or modules into groups expanded with count or for_each can now be done in a single apply step, showing the simplified view of the prior and new resource names. Intermediate steps that resources may have passed through don’t need to be displayed, nor do changes that did not affect existing resources at all.

  • Refactoring within Terraform Cloud workflows: Refactoring based on the configuration itself allows all changes to be evaluated and applied from a plan. Cloud practitioners can use the same workflows they are accustomed to for refactoring operations that were not previously possible.

  • Enabling module authors to refactor a public module: An author of a shared module can safely refactor the internals of the module without requiring manual steps be taken by the practitioner. The moved blocks can be committed in the module source, and only when existing state with the previous resources are encountered, will the necessary move operations take place.


How to try it out

Pre-reqs

  • Install Terraform v1.1.0-alpha20211006
    • For help installing multiple versions of Terraform, many community members use version management tools like tfenv or tfswitch.

Recommended setup

  1. For tutorials and use-cases, the documentation is the best place to start.
  2. For suggested environment setup, read on below.

We suggest first writing a small configuration that calls your real module, in a version control branch forked from the latest stable release of your module. For example, you could make a subdirectory refactoring-test inside your module’s directory and instantiate like this:

module "main" {
  source = "../"

  # (and then some input variable values that will
  # make sense for a standalone test environment)
}

If you run terraform init and then terraform apply in this refactoring-test directory, you can then create a baseline state representing your module as currently implemented.

Once that’s successfully created, you can try editing your main module’s source code to implement the refactoring you want to try, including any moved blocks to describe the changes you’ve made. Once you have at least one moved block in your module you’ll also need to enable the config_driven_move experiment within your module, which you can do by setting the experiments argument inside a terraform block inside your module directory:

terraform {
  # "config-driven move" is the project name for our first
  # set of work related to refactoring, focused on moving
  # objects between resource instance addresses. This
  # subset of the functionality will be stabilized for v1.1,
  # but we hope to continue adding other refactoring
  # features in later releases.
  experiments = [config_driven_move]
}

With the configuration changes made, run terraform plan to see the effect. If it worked, you should see Terraform report that some existing objects from the state moved to new addresses. If your moved blocks had from set to an address created by original terraform apply then you should see that it overrode Terraform’s usual behavior of planning to destroy the object at the old address and to create a new object at the new address.


Intended audience for Alpha feedback

Note: This feature is a bit more challenging than most to try out in a development environment, because it relates to ongoing changes to an already-existing instance of a module. We recommend instantiating your updated new module against an empty Terraform state, so these features won’t have an impact.

  • Maintainers of shared modules: Specifically shared modules that are used by multiple callers, especially public modules in the Terraform Registry. It’d be great if you could think of some refactoring you’ve either done in the past, or hope to do in the future – and try implementing those changes in a test-only environment.

  • Terraform Cloud/Terraform Enterprise users: If interested, please reach out to me (kalpers@hashicorp.com) for more details on how to test this. At the moment, we recommend trying this out with local state only, and setting up local credentials. We also can help ensure this isn’t used in production.


How to share feedback

  • For this first round of feedback, please comment directly on this post. At this early stage, we’re aiming to keep all feedback and bug reports separate from those in the current release of Terraform. That being said, we’re looking for all failure and success stories – all feedback is valuable to our work.
  • As we get closer to the stable release of these features, we’ll close this topic and ask for feedback as separate issues in GitHub.

Scope and known limitations

This first set of functionality also intentionally has a limited scope so that we can solve a subset of problems now, rather than waiting for a perfect design for all possible problems. In particular, the following are intentionally out of scope for this first iteration:

  • Moving objects between states/configurations. These features are only for moving objects between addresses in a single state/configuration.

  • Moving objects between modules packaged separately. The first iteration only allows moving objects between modules that are distributed together in the same repository or other module package, to avoid creating a new kind of external compatibility constraint for shared modules. We want to gain experience with the process of refactoring within a family of modules maintained together by the same team first, before trying to solve the more complicated problem of transferring responsibility to other module packages possibly maintained by other teams.

  • Dynamically-decided source and destination addresses. The first iteration focuses only on the more straightforward case of moving between two addresses we can write down statically in the configuration.

If you have modules where you need the out-of-scope capabilities described above, we’d still love to hear the details about your use-cases if you’d be willing to share them. However, we want to be explicit that we don’t intend to address that particular sort of feedback until a later release.


Thanks in advance for your testing efforts and feedback! If you’d also be interested in jumping on a call after trying it out, feel free to comment below and I’ll follow-up with you. We really appreciate your time on this, and excited to see how you like it.

15 Likes

Hi Korinne and all.

I’m an author of tfmigrate which is a third-party tool for state migration inspired by database migration. I’m glad to hear that the Terraform core team is working on the refactoring feature. I believe that it’s a missing piece for Terraform.

I played with an alpha build and confirmed that this feature would be useful for many practitioners, especially for module maintainers. I like the design philosophy that state migration is natively integrated into a normal plan & apply workflow. This allows module maintainers to refactor the implementation details without asking module users for additional work. When the change is only implementation details, I agree that the migration file should be provided by the module maintainer, but it has been hard to support by tfmigrate because the module maintainer doesn’t know a full resource address of module instance. Other areas that the current implementation of tfmigrate doesn’t support well are workspace and terragrunt. I expect that the native refactoring feature would also make these users happy.

Knowing that it’s out of scope for the initial release of this feature, but let me share another use case about refactoring, that is, refactoring directory structure. I originally wrote the tfmigrate to refactor my directory structure for Terraform configurations, which needs splitting a large state into multiple small states. As you know, If you want to change the directory structure, it typically needs moving resources across state file boundary. I understand it’s out of scope for the initial release, it’s ok, but I would like to emphasize that it’s an essential feature for practitioners who running terraform in production for a long time. Because the optimal directory structure depends on many parameters: the characteristics of system, the structure of organization, the frequency of changes, the skills of your team, the current technical constraints of terraform core and ecosystems and so on. In addition, they are actually not fixed parameters, they will change over time. My today’s best practices may easily become future technical debt. There is no silver bullet for the best directory structure. All we need is a way to refactor it easily without production outage.

As you know the current architecture of Terraform CLI assumes that it handles one state except for state mv. You might think that we can move resources between states by combining state rm and import command, but it doesn’t work well because an argument of import command depends on resource type and provider implementations. Some resource type doesn’t have a unique identifier of real resource, they have special syntax for import (eg. aws_iam_role_policy_attachment, aws_security_group_rule, etc.) To make matters worse, these rules are implementation details of the provider and cannot be obtained from the provider’s schema. This means that we cannot automatically generate import command. This could be resolved by modifying the provider sdk and standardizing the import interface. It’s not easy but not impossible. The worst problem is that some resource types have an attribute which cannot be retrieved from API (e.g. aws_s3_bucket.force_destroy, aws_db_instance.password, etc.) In such case it’s simply impossible. Aside from refactoring, standardizing the import interface makes sense, but for moving resources between directories, I think using rm and import doesn’t work well.

I have no idea what will be the best way to work with multiple directories in Terraform CLI natively, but I shared the issue I’m aware of. I understand that it cannot be resolved immediately, so I’m looking forward to the initial release :smiling_face_with_three_hearts:

Thanks!

7 Likes

Thank you for this very thoughtful feedback. I’m glad you found this first implementation useful, and appreciate the time you took to try this out.

The use-cases around refactoring directory structure are something we are actively discussing, as we know it’s a sticking point for Practitioners. It’s definitely a priority for us as we move forward.

Thank you again :blush:

1 Like

@korinne the docs links in the initial post seem to be 404-ing for me - are there updated links to the docs for this feature?

1 Like

Thanks for pointing that out. I’ve restored the deleted branch so the doc links in the first post should work again. We’ll update the links later.

2 Likes

@korinne this sounds like a nice feature to have for our day-2 operations.

When using my custom build tool aztfmove, I’d like to reimport resources to reset resources after a move on Azure. Similar to moved, an import option based on a supplied ID would make that operation more integrated.

That might be out of scope for now, but as it resides in the same corner I’d like to at least raise it :grimacing:

1 Like

Hi! That’s a great suggestion, thank you for that use-case. While it is out of scope for now, I’ve noted it in our backlog for future updates.

I didn’t know this was coming, but it has made my day. My number one complaint as someone who works with Terraform day in and day out is that state management is overly difficult and I believe this will move us in the direction where that gets a lot better.

Other than to say I’m stoked on moved coming to v1.1, I’m commenting primarily to echo @minamijoyo’s request: We need a better way to move resources across different state files. Breaking up terraliths is a common job I get put on as a Terraform consultant and I find it to be a painful, tricky procedure that my clients don’t typically understand. This initial feature work seems like it could provide the full path forward in that regard, but it will need to be realized. Maybe v1.2? Regardless, I look forward to that day!

Thanks for implementing something we’ve been asking for folks. I’m excited to use it and see how this pans out!

1 Like

Thanks for the alpha feedback, everyone!

Now that v1.1.0-beta1 is out we’re switching into general prerelease feedback mode, and so I’m going to lock this topic. We’d still love to hear any feedback you have about this feature if you try it during the beta period, but please now send that feedback as GitHub issues so that we can track it all in one place as we prepare for the final release.

Thanks again!