Terraform Core: Current Research projects

Terraform
,
1

Hello all :wave:,

My name is Korinne, and I’m the Product Manager for Terraform Core. As you might’ve seen, we’ve been intermittently requesting feedback on specific features (such as the v1.1 moved statements and having the ability to raise an error), and have received really helpful examples and use-cases.

I first want to say thank you! We are very grateful for all your comments and feedback, and we wanted to share an update on the other projects we’re currently researching. At this time, we can’t make any commitments about when these projects will come to fruition, but wanted to be transparent about some of our popular requests.

With that, here are some topics we would love to get your feedback on, including specific use-cases. We’ve seen many discussions throughout community forums and issues about these topics, but thought it would also be helpful to formally request feedback.

Current research:

  • terraform test (Module Testing Experiment). This is an experiment that we’d like to get formal feedback on, whether it’s feedback on the syntax/test configuration itself, or further use-cases you’d like to see supported. If you also have ideas of how to incorporate the recently released preconditions and postconditions (as part of the v1.2 alpha ), we’d love to hear about them.

  • Improvements to the new moved blocks, as well as feedback on ways to improve refactoring tasks overall. These were recently released in Terraform v1.1, and we are hoping to get more feedback on different ways to expand their functionality, as well as other needs around refactoring/ongoing maintenance overall.

How to participate

If any of these topics are something you’d like to see in a future release, or have use-cases in mind, we’d love to hear from you! You can either comment directly here, or sign up for a call to discuss further on my Calendly link. If you schedule a call, please let us know in the notes which topic(s) you’d like to discuss.

A call is always ideal, as we can go into your background a bit more and talk about your workflow. However, if you’d rather just comment, it would be helpful to know:

  • What is your role?
  • What improvements/requirements would you like to see delivered? What problems do they solve?
  • In the absence of these feature improvements, do you have any existing workarounds? What are they?

It should be noted that these are not all the research projects we’re currently working on, but ones that require more specific use-cases to flesh out the solution. Thank you in advance, and excited to hear how we can improve Terraform!

3 Likes
2

Hi Korinne, thanks for this topic.

The main idea of testing is to pass different inputs and validate results. Current implementation looks more like validation, instead of testing.

Let’s say we have a simple module with simple logic

resource aws_instance "instances" {
  count = var.count
}

If we want to test this logic, we should pass 0,1,2,3,4 and check how many instances will be created (or planned for creation)

This simple case can be covered by using outputs.

output "api_base_url" {
  value = aws_instance.instance

  precondition {
    condition     = count(aws_instance.instances) == var.expected_count
  }
}

Still, it will require to define multiple outputs or null_resources just for tests purpose.

Also, usually complex logic is hidden in local variables and it can be tested before plan (to save time). It’s not clear how it can be tested for now.

I’m recommending to use best practices from programming languages.

As example:

resource aws_instance "instances" {
  count = var.count
}


# I'm using test_plan instead of precondition for better clarity
resource test_plan "aws_instance" {
  scenarios = [
    {
       vars = {count: 1}
       conditions = [count(aws_instance.instances) == 1]
    },
    {
       vars = {count: 3}
       conditions = [count(aws_instance.instances) == 3]
    },
  ]
}

Same resources can be created for testing apply and for testing local variables rendering (if those variables didn’t require plan\apply)

wdyt?

1 Like
3

They’re definitely a step in the right direction, but the implementation in 1.1 is too limited to be of much use for the kinds of issues I’ve run into.

For context, I’m primarily using Terraform to manage a large complex Vault configuveration. Terraform runs happen inside a deployment of Terraform Enterprise. For reasons of security, I’m not allowed to see my own Terraform state file, since it would contain administrative credentials that no human is ever allowed to see - so automation is key to getting any refactoring done.

As a result, I’ve ended up building automation that allows submitting parameter lists for commands like terraform state rm, terraform state mv, terraform state replace-provider and terraform import, and runs the commands in a locked down environment where a privileged TFE API token is available.

I have some scenarios that might be interesting to you:

Complex move scenario

I have a Terraform configuration which defines a lot of Vault namespaces.

The top level Terraform configuration is one module block per Vault namespace, pointing at a subdirectory for each namespace. Each of those namespace subdirectories in turn references additional modules which set up various different parts of the Vault configuration. So the Terraform resource addresses end up looking something like this:

module.namespace_one.module.ns_basics.<some resources>
module.namespace_one.module.ns_approle.<some resources>
module.namespace_one.module.ns_policy.<some resources>
module.namespace_one.module.ns_ssh.<some resources>
module.namespace_one.module.ns_other.<some resources>
module.namespace_two.module.ns_basics.<some resources>
module.namespace_two.module.ns_approle.<some resources>
module.namespace_two.module.ns_policy.<some resources>
module.namespace_two.module.ns_ssh.<some resources>
module.namespace_two.module.ns_other.<some resources>
<repeat for hundreds of namespaces>

The original design choices used a lot of separate modules, but it turns out managing resource dependencies across module boundaries is somewhat cumbersome, and if we could simplify by combining some of these building-block modules into fewer, we would.

But, v1.1 moved blocks can’t help us here because they only support moving resources within a module, or deeper into nested modules which are sourced from a local path, and this is neither of those.

Import scenario

We had an existing production Vault instance, and had to bring existing configuration objects under Terraform control. We managed it because terraform-provider-vault and Vault APIs mostly don’t distinguish between creating an object and updating it, so we were able to have Terraform believe it was creating resources whilst it was actually just updating them with a copy of their current configuration.

However, Vault identity group aliases don’t work this way, and we had to use terraform import for those - which we had to automate ourselves. Terraform could use a new kind of resource meta-argument in its configuration language to enable it to execute import operations prompted by the config file.

The reverse is also true. It would be helpful to be able to tell Terraform to forget about objects from its state, without actually destroying them, via another meta-argument.

Refactoring provider aliases scenario

We have several modules using poorly chosen provider alias names. Currently Terraform has no way to allow me to correct this, short of building something that will directly edit the state JSON myself.

Let me know if you’d like a call to further explore these.

4

Thank you for your response, this is really helpful. Agreed completely – the preconditions and postconditions functionality isn’t intended to be a true “testing” solution, but a complementary way of validating inputs and creating a cleaner software contract. With that, would you be open to talking further about some more complex logic you’d like to be able to test with Terraform? If so, you can book some time on my Calendly link . Thank you again!

5

Hi, this is really valuable feedback – thank you! All of your suggestions definitely resonate with us, and I’d love to talk further in a call if you’re open to it. If so, feel free to grab a time on my Calendly link and we can chat more. Thanks again!