Cloudformation to Terraform migration along with gitlab with 5 stages to trigger seperately

Terraform
1

We are migrating our infrastructure automation from Jenkins + CloudFormation to GitLab CI/CD + Terraform and need guidance on the best approach.

Current Setup:

  • A GitLab CI/CD pipeline (gitlab-ci.yml) acts as a parent pipeline with five manually triggered stages (Application, DB, S3, Undeploy, Fallover).
  • Each stage triggers a child pipeline that executes shell scripts stored in a helper-scripts folder.
  • These scripts reference CloudFormation templates (cloudformation folder) and parameter files (cf-parameters folder).

Migration Goals:

  1. Convert CloudFormation templates to Terraform while maintaining modularity and reusability.
  2. Refactor the GitLab pipeline structure to align with Terraform best practices.
  3. Manage input parameters and state files efficiently.

Questions:

  1. How should we structure Terraform files and modules for this transition? Should we keep a similar directory hierarchy or reorganize it?
  2. What’s the best way to pass parameters dynamically from GitLab CI/CD to Terraform (similar to our current JSON parameter files)?
  3. How should we handle Terraform state when multiple pipelines (child jobs) need to update the infrastructure?
  4. Are there any best practices or pitfalls we should be aware of in this migration?

We’d appreciate any guidance, best practices, or examples from similar migrations.

1 Like
2

Main suggestion would be that I would also look at open source or commercial tools (Atlantis, Spacelift, Hashicorp Cloud) that implement this (vs. trying to use GitLab or other generic CI tools).

That said, at some limited scale, you can probably do this; if you do go this route, I’d suggest creating a generic workflow and run it once per state (using a different working directory). I think there are some simple examples online.

1 Like
3

For your migration, structure Terraform with modular directories (e.g., modules/network, environments/dev). Use GitLab CI/CD variables and .tfvars.json for dynamic parameter passing. Store Terraform state in AWS S3 with DynamoDB for locking to prevent conflicts. Implement workspaces or separate state files per environment. Best practices include using terraform plan, enforcing state locking, and maintaining reusable modules. Watch for state drift, improper variable handling, and rollback challenges.