Migrating to a Multi-repo approach

HI friends, i have been using and introduced terraform to a startup i work at. I have been using terraform for about a year now and our infrastructure has grown since i first started using it. Our company has also grown and i will not longer be just an Individual contributer, now i will have other team members who will get involved in infrastructure changes. And so we need to make our terraform configuration scalable.

I’ve been doing some research into the multi-repo approach and i like the design and think it would be suitable for us.

I have some concerns about migrating what we have to a multi-repo approach and would like some guidance and confidence that it would be possible.

- Current Setup

In our current setup we have a single repo, with a set of folders for our infrastructure needs. We store the state file for each repo in a single s3 bucket e.g s3://acme-terraform-state/env:/prod/rds/terraform.tfstate.

infrastructure (repostiroy)

  - rds (folder)
      - main.tf
  - eks
  - mysql
  - mongodb

- Multi-Repo

We will migrate the folders to their own repo and prefix it with tf-module

tf-module-rds

main.tf

Then we will create another repo which will determine what is currently live (imports the module).

infra-live-rds

 - prod
    main.tf
 - dev
    main.tf

How do we migrate the existing state file to fit this new multi-repo approach. We can not afford to recreate infrastructure resources.

Thanks for linking to the blog article, to explain what you meant by “multi-repo approach”. (It’s not really a standard term outside of that article, in my experience.)

The biggest thing I’d like to emphasise in response to your post, is this is not an all or nothing decision. I fully support the idea of storing re-usable modules that are separately versioned, and included in multiple other configurations, in their own Git repositories. On the other hand, if you have modules that are used exclusively in only one other repository, I would strongly question what anyone seeking to split them out was hoping to gain from that - and have merged Git repositories to undo that sort of separation.

In order to give further advice, I think you’d need to share with the forum a bit more detail.

There’s a colon in an odd place in your S3 URL - is that colon after env supposed to be there?

You said you have a state file for each repo, but the S3 URL, and pictured repository layout seem to contradict this - it implies you have a state file which is:

  • specific to each environment (prod)
  • specific to the top level directory within the infrastructure repo (rds)

You’ve not mentioned how your environments (e.g. prod) are represented in your current setup - please elaborate on that.

That might be a good idea - or it might not be. Which it is, depends a great deal on whether this module is a clearly thought out re-usable unit, or just “everything my project needs to do with RDS”.

It also matters quite a bit how you use your prod and dev environments. Do all your changes to prod, first pass through dev, like a flowing pipeline? Or, are there likely to be things in dev which are experiments that will never go to prod? What about emergency changes in prod that don’t necessarily go through dev?

What do you anticipate living in this repository? Just a tiny configuration with a single module block identifying a version in the tf-module-rds repo? Or something more complex?

At present I don’t feel I understand your environment well enough to have a clear opinion, but I have a hunch that if you share more, I am likely to end up thinking you’re likely to hurt yourself more than help, by going through with this migration as currently outlined.

In order to complete a migration like this, you will need to develop an excellent understanding of Terraform moved blocks. It will quite possibly be useful to practice, by creating some sample configurations using null_resources, and trying out the kind of migration you aim to implement. You’d then practice creating your migrated configuration, adding moved blocks and running terraform plan against your existing state, to validate whether you have successfully mapped everything from your old layout to the new.

In order to give further advice, I think you’d need to share with the forum a bit more detail.

Thanks for your response, i will try my best to go further in detail.

Thanks for linking to the blog article, to explain what you meant by “multi-repo approach”. (It’s not really a standard term outside of that article, in my experience.)

Suprised by that, i have seen it mentioned in a few different places. What is the standard term for this approach?

There’s a colon in an odd place in your S3 URL - is that colon after env supposed to be there?

You said you have a state file for each repo, but the S3 URL, and pictured repository layout seem to contradict this - it implies you have a state file which is:

  • specific to each environment (prod)
  • specific to the top level directory within the infrastructure repo (rds)

You’ve not mentioned how your environments (e.g. prod) are represented in your current setup - please elaborate on that.

The url was copy and pasted directly from aws s3 console, it does contain that :

You are correct our state file is specific to a terraform workspace and the specific top level directory.

I did not think mentioning that we use workspaces was worth it, as to be honest its more of a “habbit” than anything else, after the init process we select a prod workspace terraform workspace select prod.

We use it for “some” of our infrasturcutre to build a dev version of the resource, before prod to test new configurations. But we found workspaces are difficult to deal with when it comes to seperating environments. Its also one of the main attractions i see with “multi-repo” approach.

That might be a good idea - or it might not be. Which it is, depends a great deal on whether this module is a clearly thought out re-usable unit, or just “everything my project needs to do with RDS”.

The tf-module-rds will be a reusable unit to create aws rds instance including the cloudwatch alarms etc all built in terraform.

It also matters quite a bit how you use your prod and dev environments. Do all your changes to prod, first pass through dev, like a flowing pipeline? Or, are there likely to be things in dev which are experiments that will never go to prod? What about emergency changes in prod that don’t necessarily go through dev?

We test configurations in dev before prod. But we rarely do this, i anticipate this becoming more relevant in the future.

What do you anticipate living in this repository? Just a tiny configuration with a single module block identifying a version in the tf-module-rds repo? Or something more complex?

Exactly, i anticipate prod/main.tf to have a single module block identifying the version of tf-module-rds repo. Also having a dev/main.tf to have a module block identifying a different version of ts-module-rds repo. Allows us to tset changes and see a clearer seperation than what we have with using terraform workspaces.

The biggest thing I’d like to emphasise in response to your post, is this is not an all or nothing decision. I fully support the idea of storing re-usable modules that are separately versioned, and included in multiple other configurations, in their own Git repositories. On the other hand, if you have modules that are used exclusively in only one other repository, I would strongly question what anyone seeking to split them out was hoping to gain from that - and have merged Git repositories to undo that sort of separation.

You are correct a lot of these modules will only be used exclusively in only one other repository. But i believe

  • having things separately versioned is the main attraction via tagging modules.
  • being able to build ci /cd pipelines to automate and verify infrastructure changes other team members are making will be much easier if we have multiple repo’s,
  • i can assign users specific permissions to modify one repo but not the other, if everything is togeather, all of our devs will have access to every infrastructure config.

My impression is that there isn’t one and doesn’t need to be, because leaning hard into every module will be in a different repository, or every module will in one repository, is unnecessary, and it’s really something you can flexibly opt one way or the other on a case-by-case basis.

Aha, understood. So you use workspaces to point the same directory in Git at several different state files. And then, you presumably also have some other method or wrapper, to point Terraform at different infrastructure environments? (e.g. -var-file or environment variables)

But, is it actually attractive, to have to look at one repo to get a (presumably autogenerated, automatically incrementing) version number, and then cross-reference a separate repo to see the contents? As compared with just looking at the Git history directly?

These are good reasons to separate the rds code out of the single infrastructure repo, but don’t necessarily motivate going all the way to one repo containing the code and tags, and another module selecting a tag.

You should also consider whether you will get the feedback you desire from your CI/CD, if changes to a module don’t get evaluated in the context of actual infrastructure until they have been committed, tagged, and the version bumped in another repo.

This may possibly lead you to decide that tagged releases are desirable for prod but not for dev, for example.