I’ve got a scenario where I’ve got a root module leveraging a RDS creation module.
I am using it to deploy RDS instance and the security whitelisting all configured with a envname.tfvars
As I scale this, I see that I may need to provision 100 different environments as it’s a “platform as a service” offering.
Based on my reading for best practices with terraform, I’m seeing I’d likely have the entire repo copy/pasted for each environment all pointing to the module. I see that workspaces were not designed to provide the environment seperation needs i’d like need for this scenario. Is there any better way to do this? Do I basically need to adapt the module to be the single point of creation and each folder gets it’s own inputs for each environment, or is there any method using just different tfvars for each that I can avoid a lot of copy/paste?
I’ve considered leveraging terragrunt, but before doing this, figured I’d ask how any one else scales the plan as most examples talk about qa/uat/prod based examples, while I’m talking about
modules
--- rds
plans
--- org1
------ main.tf
------ org1.tfvars
--- org2
------ main.tf
------ org1.tfvars
That’s a lot of repeated code when I was hoping to instead just have different var files.
2 Likes
Hi @sheldonhull,
Without some specific examples to work with it’s hard to give specific advice, but as you’ve seen there are two common ways to represent multiple environments:
-
If all of the environments have essentially the same “shape” but differ just in size and number of objects, a single configuration with multiple .tfvars files can be reasonable, although it has the downside that you need to manually select the right variables file for each environment when you apply. That’s why we usually recommend the other approach instead…
-
In practice, environments commonly have differences that are more convenient to represent in full Terraform configuration rather than just variables. In that case, we recommend to use one configuration per environment where each configuration’s root module consists only of a few module blocks that instantiate common building-blocks but perhaps wire them together in a slightly different way to represent the differences between the environments. This is an application of module composition, which is a more general idea that is the primary way to represent higher-level infrastructure “components” in Terraform.
As systems grow, there arise other opportunities to smooth out the connectivity between components. For example, if you split your infrastructure across multiple separate configurations per environment – for example, to separate infrastructure by tier or by change frequency – it can be convenient to use a hybrid approach:
- You can establish each environment with a top-level configuration using the second approach above, but define an environment as having, at its core, some sort of configuration (e.g. AWS SSM Parameter Store, HashiCorp Consul, etc) store that contains information about that environment. Other components in your infrastructure can then be built using a variant of the first approach, where a single configuration is deployed multiple times, but in this case rather than passing
.tfvars files you can instead use the environment’s configuration store to access per-environment settings. In this case it can be reasonable to use Terraform Workspaces to represent that switch, because the configuration store encapsulates all of the shared data for each environment and so switching workspace is all that’s required to select the appropriate settings.
The main reasons we recommend multiple configurations over a single configuration with .tfvars files are:
- Running Terraform is more straightforward: you can just run
terraform plan and terraform apply without worrying about appying different arguments, making sure you select the right variables, etc.
- In practice environments tend to have, at their core, differences that are easier and clearer to model using the full Terraform language (via module composition) rather than via complex conditional configuration based on variables. For example, production environments tend to have stricter access controls than pre-production environments, and tend to be wired up differently to cross-cutting concerns like monitoring and tracing systems.
Terraform is designed with some flexibility so that you can use it for certain less-typical purposes, but by default we recommend starting with one configuration per environment and then, if necessary, expanding into the hybrid model described above with multiple configurations per environment with a shared configuration store.
3 Likes
Adding a follow-up after successfully applying this.
This worked pretty well. I think the main thing I’d like to improve now is my settings inheritance.
I basically setup the following
orgs
├── prod
│ ├── tacobear
│ ├── local.override.yml
│ ├── burritos
│ ├── local.override.yml
│ ├── terraform-cloud-workspaces
| └── prod.settings.yml
└── qa
| ├── testabilitytacos
│ ├── local.override.yml
| ├── terraform-cloud-workspaces
| └── prod.settings.yml
I setup all the default in yaml for easily managing and editing. I basically pull the default values from the one level up, and then do a
locals {
settings = merge(local.default_prod_settings,local.override.yml)`
}
While this has worked pretty well, yaml decoding adds another potential layer of complexity.
Based on rereading the documentation, would it not be better to instead setup do replace the prod.settings.yml with something like a locals that gets merged with another locals file, or leverage the settings.autos.tfvars to set the values?
I want my global general settings for the prod or qa environment to be able to be edited in single place and then basically overridden as needed. Since I’m running in Terraform Cloud I also don’t think using a tfvars file is recommended, or an overrides file itself.
How would you approach this to best use inherited defaults one level above a plan, while still making it easy to override anything needed in the individual plans? While my solution works, I’d like to simplify further with any improvements i make.
As a quick fix I just tried using a local.tf file and pulling content in but I don’t think that will work without using symlinks. I’m really adverse to using symlinks for this… seems to add some accidental complexity and would prefer to use better approach.
In one case I want to build a default whitelisting for a security group and then append additional entries from a child folder, and in other cases I want to merge to ensure settings can be overriden.