Migrating existing terraform structure to terraform cloud

Terraform HCP Terraform
1

Hello Team,
We have looking for terraform cloud for our existing TF structure. Below is the high level structure,
Total two repositories

  1. We have different modules in one repo under subdirectory
  2. Environment specific TF structure (dev,staging, prod) in another repo.
  3. Creates VMs on on prem vsphere and AWS
  4. Secrets are fetched from AWS secret manager
  5. Install app by chef once provisioned

Questions are

  1. What is the best workspace approach in such scenario API driven, CLI driven or version control one?
  2. Whether from TF cloud, how connectivity will happen to vsphere or AWS infra to install app through CHEF? Any conf needed on this?
  3. Is there any option to run AWS CLI to fetch AWS secret and use during init/plan? If not, what is the best option?
  4. Most of the tfvars files contains below variables, how to pass such variables to TF cloud?
    e.g.
    vm_info = [
    {
    “hostname” = “abc”
    “subnet” = “subnet-abc”
    “private_ip” = “1.2.3.4”
    “instance_type” = “c5.4xlarge”
    },
    {
    “hostname” = “xyz”
    “subnet” = “subnet-abc”
    “private_ip” = “1.2.3.5”
    “instance_type” = “c5.4xlarge”
    }
    ]
2

Hello @pravinksavant.

Questions are

  1. What is the best workspace approach in such scenario API driven, CLI driven or version control one?

/R That really depends on what you want to accomplish and your CI/CD strategy. For example, If you are going to be recreating environments continuously while testing changes to the code, the VCS workflow leveraging branches could be a good alternative.

  1. Whether from TF cloud, how connectivity will happen to vsphere or AWS infra to install app through CHEF? Any conf needed on this?

There are a couple ways to tackle this one and lots of “depends” too. You can leverage the remote-exec provisioner to connect to the machine where you are running CHEF from.
Things for consider for connectivity:
-Security group / Firewall rules allowing TFC workers OR TFC Agents traffic
Terraform Cloud Agents - Terraform Cloud and Terraform Enterprise | Terraform | HashiCorp Developer
IP Ranges - Terraform Cloud | Terraform | HashiCorp Developer

  1. Is there any option to run AWS CLI to fetch AWS secret and use during init/plan? If not, what is the best option?

Is the AWS Secret is Static. Adding it to TFC with the Sensitive checkbox checked is an option. Once set on TFC they cannot be seen again.
Workspace Variables - Terraform Cloud | Terraform | HashiCorp Developer

  1. Most of the tfvars files contains below variables, how to pass such variables to TF cloud?
    e.g.
    vm_info = [
    {
    “hostname” = “abc”
    “subnet” = “subnet-abc”
    “private_ip” = “1.2.3.4”
    “instance_type” = “c5.4xlarge”
    },
    {
    “hostname” = “xyz”
    “subnet” = “subnet-abc”
    “private_ip” = “1.2.3.5”
    “instance_type” = “c5.4xlarge”
    }
    ]

For this one, you can either have a terraform project using the TFE provider to create the variables on the desire workspaces. Also, Variables Set is a new feature that allows for common or shared variables to be set once but and be available for all or selected workspaces.
Anther method is aving a *.auto.tfvars file in the root of your terraform project. TFC will pick this up and use those values.
Workspace Variables - Terraform Cloud | Terraform | HashiCorp Developer
TFE Provider - Terraform Registry

Hope this helps a bit,

Simon

3

@sgarciajaramillo Thanks for detailed information.
For point 1 - We have decided to go with VCS workflow approach.
For point 2 - we are thinking for self hosted agent in our vmware and AWS environment
For point 3 - We are thinking on either go with env variables with sensitive or vault
For point 4 - We are thinking to put all these in auto.tfvars.

We have started designing workspaces.

Another small doubt here,
We have few modules defined in a separate directory each as part of single repo in bitbucket. e.g terraform-module. Now, when I publish this to registry-modules section, I don’t see these are published.
So question is, do we need separate dir for every module we publish or it can be part of single repo?
If yes, how that can be referred in workspace then.
e.g.

module “modules” {
source = “app.terraform.io/org/modules/aws”
version = “7.0.0”
}

How to refer the submodules under single dir repo?

4

@pravinksavant

Great update, certainly encourage you to use the Vault integration.

To get your modules published to you Terraform Cloud Registry, they need to be on a separate repo in your VCS. You will tag the commit to differentiate the release.

This is the recommended approach for modules so they can be shared/used by other workspaces within your terraform cloud organization.

Best regards,

Simon