Terraform and git trunk based deployments

I just wanted to get some feedback on how others are managing the deployment of terraform in different environments (eg , dev, test , Prod). There will be a variation between the environments as the QA testing can take around 2 weeks.

I’m planning on using this method

Scaled Trunk Based branching

  • Develop directly on the main branch (Trunk branch)
  • The main branch (trunk branch) is the single source of truth for the project
  • Feature Branches will be used for code review (only) before committing to main
  • PR to approve merge to main
  • No release branches
  • Deploy direct from Main for all environments

Our project infrastructure code is packaged into one root module, this root module will be built using smaller local sub modules. This will simplify the control of versioning between environments.

We will refer to the modules in the following way :

module “Test” {
source = “github.com/org/repo//modules/application/ref=v3.0.0

}

module “Prod” {
source = “github.com/org/repo//modules/application/ref=v2.0.0

}

etc

To simplify the versioning I will make an application root module , and refer to that.

This seems to be the simplest method but the team would like to keep everything in one mono repo module rather than 2 repos as I suggested.

I can’t see a way of doing this without using git branches , which will over complicate the deployments. I would be interested in how others have approached this.

Thanks

You want to have a single Terraform root module, which incorporates dev, test, prod as module blocks?

Personally, that sounds like a mistake to me.

It commits you to having a single Terraform state file that encompasses all of your environments, rather than one per environment, which means that:

  • You’ve sacrificed the ability to test anything that operates at the level of the Terraform state in lower environments. You want to move to a newer version of Terraform? Can’t deploy to dev first, it takes effect on everything including prod at once. You find you need to perform some manual manipulation on your Terraform state? Can’t test it cleanly on a lower environment first, as all of your environments are in one state.

  • You’ve sacrificed some scalability - Terraform doesn’t deal well with huge states, and you’ve applied a multiplier to yours by combining all environments in one.

  • You’ve sacrificed concurrency - if a Terraform run is working on deploying a change to one environment, you have to let it finish before it will be able to start doing something else.

  • You’ve introduced unwanted coupling between environments - let’s say, one day, an engineer wants to push some minor change to dev, and the Terraform plan flags up that it intends to make some unexpected change to prod - maybe there has been some manual change made outside of Terraform control, which Terraform wants to revert, or maybe it’s a bug. Whatever the reason, either your dev pipeline is now blocked whilst you diagnose prod, or the change goes unnoticed and rolls out without the oversight appropriate to production changes.

In conclusion to that part:

You absolutely want separate root modules per environment.


Exactly how you model this in Git is up to you.

I would be hesitant to use a monorepo, because I regard it as misleading: Git tags are inherently for the whole repo, so if you tag versions, you have to have internal team documentation which explains “Oh, our tags relate to these sub-paths within the repository, but not those other sub-paths”.

Whether you use Git branches for each environment, or separate subdirectories within the same branch instead, is up to you. There are trade-offs with each:

  • With branches, you can’t update all environments in one PR - cross-cutting refactors are harder.
  • With directories, you can’t merge between them, and are reduced to manually copying the changes, and re-reviewing them.

Lastly, consider that whilst deploying tagged versioned releases makes sense for prod, you almost certainly do not want to set up your dev environment such that every Terraform change needs to be merged to the main branch, tagged, and have a version number updated in another file, before it ever gets to dev.

This may involve a Terraform configuration that fetches the application module directly from a Git branch, in the dev environment (but then, think carefully about how and when Terraform runs will be triggered, based on which paths in which repositories will be changing.)

Thanks for your reply .

This is very good point , and it makes be believe the 2 repo structure is the correct way to go:

I would be hesitant to use a monorepo, because I regard it as misleading: Git tags are inherently for the whole repo, so if you tag versions, you have to have internal team documentation which explains “Oh, our tags relate to these sub-paths within the repository, but not those other sub-paths”.

I didn’t make it clear about how the structure will work :

I will have an application module that contains the all the infastructure code ( one repo).
I will have another repo with the calling code broken in to folders dev test and prod , each environment will call a particular tagged version of the application module and have a separate state file each.

I may need to consider how i trigger the terrafrom runs, im using this so i have full control over what get deployed to each environment.

module “Test” {
source = “github.com/org/repo//modules/application/ref=v3.0.0

}

module “Prod” {
source = “github.com/org/repo//modules/application/ref=v2.0.0

}

I could leave dev unversioned but i would need to work out how to trigger this when the root module is updated.

module “DEV” {
source = “[github.com/org/repo//modules/application/]”

}

A quick and easy way might be to put the dev root module in the same repo as the application module. You could have a directory structure like:

modules/
    application/
    root-for-dev/

and trigger on any changes inside modules/.

Some might complain this is hacky. I prefer to see it as elegantly expedient :slight_smile:

Thanks again. This does seem to be the simplest method (i’ll just have to make sure it fits the workflow).

In general I think the 2 repo structure approach will work . Application Module - and Calling Code (TEST, PROD).

The team would prefer the one mono-repo , if I do this I cant refer to the github tags when calling the modules.

Using the one branch as the source of truth for all deployments is very appealing that is why in moving towards the folder approach rather than the branch approach.

I would be interested to see how others have approached this ( is using branching as messy as it looks).

We are going to test this instead to keep things simple .

Scaled Trunk Based branching

  • Develop directly on the main branch (Trunk branch)
  • Feature Branches will be used for code review (only)
  • PR to approve merge to main
  • Short-lived release branches to deploy to Test and Prod

Deploying a new release :

  • Development work committed to Main branch - deployed to the AWS DEV environment for DEV testing.
  • When ready we cut a new release branch from the main branch (at this point any previous release branches are deleted).Release branch deployed to the AWS TEST environment for User QA testing.
  • IF QA testing is successful we deploy this branch to the AWS PROD environment
  • GitHub Tag created to indicate what has been pushed to production (example : v.20230630130230.prod.deploy).