Advice on splitting configuration away from implementation

Terraform
1

Hello,

I’m trying to move to a terraform repo structure like this:

live
- dev
  - config.yaml
  - modules.tf
- stage
  - config.yaml
  - modules.tf
- prod
  - config.yaml
  - modules.tf
modules
- mod1
- mod2

With the idea being that the modules.tf files should be identical between the dev, stage, and prod folders, and only the config.yaml will be changing.

This is very inspired by this blog post: Understandable Terraform projects | by Didrik Finnoy | ITNEXT , but more broadly, the whole idea is that the config and the implementation is split apart from each other. It doesn’t have to a YAML file, it could be config.tf with a bunch of locals defined.

However, in the project I’m working on, which is configuring Auth0 as an IDP, many of the modules end up being essentially a wrapper around a single resource. While not ideal, the main pain is that to get info from the config to the module means defining pretty much all the inputs for the resource block as variables with default variables, and when I call that module, I have to guard all the inputs to the variables with try(local.config.some_setting, null).

Here is an example module (and I’ve omitted some variables too):

resource "auth0_resource_server" "resource_server" {
  name        = var.name
  identifier  = var.identifier
  signing_alg = var.signing_alg

  allow_offline_access                            = var.allow_offline_access
  token_lifetime                                  = var.token_lifetime
  skip_consent_for_verifiable_first_party_clients = var.skip_consent_for_verifiable_first_party_clients
}

variable "environment" {
  type = object({
    tenant               = string,
    tenant_friendly_name = string
    tenant_slug          = string
  })
}

variable "name" {
  type = string
}

variable "identifier" {
  type = string
}

variable "signing_alg" {
  type     = string
  nullable = true
}
variable "allow_offline_access" {
  type     = bool
  nullable = true
}
variable "token_lifetime" {
  type     = number
  nullable = true
}

variable "skip_consent_for_verifiable_first_party_clients" {
  type     = bool
  nullable = true
}

and in my modules.tf file:

module "resource_server" {
  for_each = { for x in local.resource_servers : x.name => x }
  source   = "../../modules/resource_server"

  environment = local.environment
  name        = each.value.name
  identifier  = each.value.identifier

  scopes                                          = try(each.value.scopes, null)
  signing_alg                                     = try(each.value.signing_alg, null)
  allow_offline_access                            = try(each.value.allow_offline_access, null)
  token_lifetime                                  = try(each.value.token_lifetime, null)
  skip_consent_for_verifiable_first_party_clients = try(each.value.skip_consent_for_verifiable_first_party_clients, null)
}

This feels like a lot of extra boiler-plate code that is relatively fragile - and seemingly not necessary, as the resource blocks all have sane default values. For example, if I did not define the signing_alg in the module, it would just pick a good sane default - but sometimes we do need to specify a different one - and that ‘edge case’ prompts the need for all this extra code.

At first I thought - “well, I can just get rid of the modules for those who have a single resource block, and instead use the resource blocks directly” - but I feel like this will lead to exactly the same issue (though maybe without the additional variable definitions) - I will have to map out the entirety of a resource block with guarded statements to ensure that every config will apply validly.

So then if I just give up on the config splitting idea - I abandon using for_each blocks, remove the config.yaml, and just use repeated resource definitions - how would it work with the different deployments of dev/stage/prod? Would I just be duplicating tons of code between each folder as needed? That also doesn’t sound like a good approach.

Anyway, any advice would be greatly appreciated, I’m a bit stuck! I’m not really wedded to any of this, so if I just need to change tack entirely, that’s fine!

Cheers
Luke

2

Terraform lacks any functionality for importing HCL snippets into an existing directory - which is why the author of the article you linked to had to copy their identical modules.tf into each environment.

Unfortunately, accepting this lack is just part of the cost of using Terraform - accept there will be some files that are identical copies in each environment, whether they are module calls, or just bare resources.

Make it very clear which files are synced and which are environment-specific.

Create a helper script in the language of your choice, for copying the synced files from a master copy to each environment. (Your choice, whether to have a separate master copy, or to just designate one environment as the master copy.)

Only create actual Terraform modules when there is a fairly sensible logical purpose to the module. Don’t create them just because you feel every resource should be nested in one.

Accept that extensive use of try() is normal when you have optional config sourced from YAML.

Remember that you can pass complex objects - even the entire parsed contents of your config.yaml - to modules if that makes things easier.

Once Terraform 1.3 comes out, have a look at its new features for specifying optional values deep in complex objects.

3

Thanks for the feedback - just having confirmation that it’s a known thing makes me feel better :sweat_smile: .

Yes, I don’t mind too much having to copy files between environments - and using a script seems like the safest way to do it too.

And yes, I do believe the optional values, with the defaults function (defaults - Functions - Configuration Language | Terraform by HashiCorp) should be helpful in this case.