Best Practice for Reusing with many environments

I’ve got a scenario where I’ve got a root module leveraging a RDS creation module.
I am using it to deploy RDS instance and the security whitelisting all configured with a envname.tfvars

As I scale this, I see that I may need to provision 100 different environments as it’s a “platform as a service” offering.

Based on my reading for best practices with terraform, I’m seeing I’d likely have the entire repo copy/pasted for each environment all pointing to the module. I see that workspaces were not designed to provide the environment seperation needs i’d like need for this scenario. Is there any better way to do this? Do I basically need to adapt the module to be the single point of creation and each folder gets it’s own inputs for each environment, or is there any method using just different tfvars for each that I can avoid a lot of copy/paste?

I’ve considered leveraging terragrunt, but before doing this, figured I’d ask how any one else scales the plan as most examples talk about qa/uat/prod based examples, while I’m talking about

modules
--- rds
plans
--- org1
------ main.tf
------ org1.tfvars
--- org2
------ main.tf
------ org1.tfvars

That’s a lot of repeated code when I was hoping to instead just have different var files.

Hi @sheldonhull,

Without some specific examples to work with it’s hard to give specific advice, but as you’ve seen there are two common ways to represent multiple environments:

  • If all of the environments have essentially the same “shape” but differ just in size and number of objects, a single configuration with multiple .tfvars files can be reasonable, although it has the downside that you need to manually select the right variables file for each environment when you apply. That’s why we usually recommend the other approach instead…

  • In practice, environments commonly have differences that are more convenient to represent in full Terraform configuration rather than just variables. In that case, we recommend to use one configuration per environment where each configuration’s root module consists only of a few module blocks that instantiate common building-blocks but perhaps wire them together in a slightly different way to represent the differences between the environments. This is an application of module composition, which is a more general idea that is the primary way to represent higher-level infrastructure “components” in Terraform.

As systems grow, there arise other opportunities to smooth out the connectivity between components. For example, if you split your infrastructure across multiple separate configurations per environment – for example, to separate infrastructure by tier or by change frequency – it can be convenient to use a hybrid approach:

  • You can establish each environment with a top-level configuration using the second approach above, but define an environment as having, at its core, some sort of configuration (e.g. AWS SSM Parameter Store, HashiCorp Consul, etc) store that contains information about that environment. Other components in your infrastructure can then be built using a variant of the first approach, where a single configuration is deployed multiple times, but in this case rather than passing .tfvars files you can instead use the environment’s configuration store to access per-environment settings. In this case it can be reasonable to use Terraform Workspaces to represent that switch, because the configuration store encapsulates all of the shared data for each environment and so switching workspace is all that’s required to select the appropriate settings.

The main reasons we recommend multiple configurations over a single configuration with .tfvars files are:

  • Running Terraform is more straightforward: you can just run terraform plan and terraform apply without worrying about appying different arguments, making sure you select the right variables, etc.
  • In practice environments tend to have, at their core, differences that are easier and clearer to model using the full Terraform language (via module composition) rather than via complex conditional configuration based on variables. For example, production environments tend to have stricter access controls than pre-production environments, and tend to be wired up differently to cross-cutting concerns like monitoring and tracing systems.

Terraform is designed with some flexibility so that you can use it for certain less-typical purposes, but by default we recommend starting with one configuration per environment and then, if necessary, expanding into the hybrid model described above with multiple configurations per environment with a shared configuration store.

2 Likes