Splitting out main.tf into separate folders

Terraform
#1

Our team generates a lot of resources. So many that our main.tf has just exceeded 10,000 lines long. This isn’t great for our developer experience and we were intending to try to split out the files to fix this. Our current structure is similar to the suggested Terraform of:

modules/
  modulea/
    main.tf
    ...
  moduleb/
    ...
main.tf // import modulea, moduleb in here
...

Our intention is to split up ‘main.tf’ by product, so as to get something like:

modules/
  modulea/
    main.tf
    ...
  moduleb/
    ...
some-business-area/
  vps_a
  vps_b
  login_system
main.tf // extends vps_a, vps_b, login_system

We don’t want to make all of the vps_a, vps_b, login_system to be redefined as modules. Doing this would require extensive work and renaming of many stateful resources. Is there a way to simply include these folders (e.g. vps_a, vps_b) in the same way that top level terraform files are included?

#2

To clarify, another reason we don’t want to put everything in modules is because we have a lot of interaction between existing resources. Calculating all the required outputs from each module would take hundreds of hours.

#3

Hi @popey456963,

I’m sorry if I’m misunderstanding your framing here, but it seems like you’re trying to go from one huge main.tf file to using multiple separate directories without considering the less extreme solution of having multiple .tf files in the same directory.

In case you’re not aware of this, Terraform doesn’t actually distinguish between the various .tf filenames and so there’s no reason why you need to put all of your resources in a file called main.tf; that’s just a convention that emerged for simpler modules that are focused enough in a particular problem that there’s no meaningful way to distinguish different components.

How to split things into multiple files is ultimately up to you; in different teams I’ve seen different strategies but I think the two most common ones are:

  • A separate file for each separate “service” or service-like structure the cloud vendor offers. For example, AWS users typically end up with files like iam.tf, ec2.tf, rds.tf to group their resource declarations by roughly the same boundaries as they’d be grouped by in the AWS console, thus making it easier to translate between what you see in the console and what you see in the Terraform module.
  • A separate file for each conceptual layer/subsystem of the overall system, as viewed by the team building the system. This one is harder to give a good example of because it inevitably depends on how your system is built, but a generic sort of version of it might be files like storage.tf for the data tier, web.tf for the web server and load balancer tiers, and worker.tf for a generic worker tier, if your system is one that makes distinctions of that sort.

Terraform considers all of the .tf files in a particular directory to belong to the same module, so you shouldn’t need to do anything except cut and paste the configurations you already have in main.tf into other files. Terraform should see the result as exactly equivalent to what you started with.

As you’ve seen, Terraform uses separate directories as the representation of modules, and so the requirements for separate directories are a little harder: you need to explicitly define the interfaces between the modules using input variables and output values. But given that the problem you started with was a sense that the main.tf file is too big, I’d suggest starting by just splitting it into multiple files in the same module and then see over time if there emerges a more explicit architectural boundary that would make the additional overhead of a module abstraction worthwhile.

1 Like
#4

While it might require an amount of effort I wouldn’t discount trying to split things into modules.

The whole idea of a module is to encapsulate a set of resources while defining a specific input and output API. Those modules can then be reused while hiding the implementation details behind a simpler facade.

A well designed module can be tested and reviewed in isolation, without having to worry how it fits into the overall system. Having a specified API makes it easier to see dependencies as well as reducing some types of bugs - you know it is totally safe to make changes to the code within a module without impacting on any of its users, as long as the API doesn’t change.

1 Like
#5

Also at the size you are talking about, have you looked at splitting the code into totally separate state files?

A huge single Terraform state (the number of resources rather than how many code lines or files that is) will be slow to work with (lots of resources to check during a refresh and lots of dependencies to calculate during a plan/apply) as well as preventing parallel deployment processes.

Often you can split things based on service, environment, tier, change cadence or team ownership, allowing multiple pieces to be deployed at a time, without having to centrally coordinate the process.

1 Like
#6

So, we’ve spent the past two days trying several things. Splitting up files by service (e.g. iam.tf / ec2.tf) didn’t help developer experience. We were constantly ‘context switching’ between files and it was unanimously decided that a single file was a better experience due to the ability to ctrl + f more easily. We considered splitting it into business area in the top-level and that was a lot better. When someone was working on a feature they were able to see all parts of it at once.

We did try splitting everything into modules as a later test and actually really enjoyed it. We used boundary in order to stop some duplication of attributes (‘global’ variables seem in bad taste, so we restricted it to ‘configuration’ type items that were required by every resource (e.g. account, environment, etc).

We’re likely going to take the leap into modules and whilst the initial cost is high we expect the ability to speedily release new features to make it worthwhile.

We do already have multiple state files split based on change cadence. Our other two projects are rarely if ever changed and we’re happy keeping them as large single file bundles. This part is possibly also starting to get to the point where we want to split it up, but plan / apply times aren’t too long yet!

Thanks for both of your help, was super useful!

#7

While it does take work in the long run (especially at the level of complexity it sounds like you have) modules should make things a lot easier. We find there is a lot of repetition across our systems in the way things are done (or standard practices we want people to follow) so we have various modules that encapsulate such building blocks - VPC setups, autoscaling groups, Lambdas, Kubernetes clusters, bundles of applications for our metrics infrastructure, etc.

We also use a selection of external modules (particularly Terraform AWS modules · GitHub) to make our lives easier. All our modules are versioned allowing a sensible change process and we use a dependency management solution (Renovate Bot) to keep everything up to date.

Some of our modules are reused over 60 times across the codebase of just under 5 million lines of Terraform HCL.