How to clone and existing AWS environment -New to Terraform

Hello, I am new to Terraform.
We have an existing AWS environment with S3, ECS, DynamoDB, Redshift, Lambdas, API Gateway, Data Pipeline and more. I do not yet have a full inventory.
My final objective is to have a lesser environment that matches the existing current environment.
I think the logical steps are:

  1. Ask Terraform to determine the current running state - Is this possible???
  2. Configure the infrastructure definition to mirror the existing environment (possible changes to size of objects, etc.)
  3. Perform a Plan and Apply
  4. Copy data between environments to enable functional & regression testing.

I can not be the first person that was asked to generate a lesser environment seeing how pervasive it is today for applications to only have production only set-ups.

Please help me to come up with a plan of action to address this issue. Any and all advice would be appreciated and thanks in advance…

Cheers -
Phil Orion - Orion like the constellation, not the Irish guy…

No, it isn’t - Terraform does not implement this part.

I don’t work with AWS myself, but I have heard this third-party tool: GitHub - GoogleCloudPlatform/terraformer: CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code mentioned a few times in this context.

However, even if automated tooling can, to some extent, analyse an existing environment, you should still expect to need to refactor any automatically generated Terraform code a fair bit, to get it into a state more suitable for ongoing human use.

@maxb - Thank you for the quick reply. Let me clone this repo and take it for a test drive to try and get started with identifying all the existing components.
I was able to import s3, dynamodb, etc, for the most part it works well. The code appears to be missing the “redshift” service. Is there a way to retrieve this or has anyone else experienced this limitation?

There is also a “transforming” GitHub - dtan4/terraforming: Export existing AWS resources to Terraform style (tf, tfstate) / No longer actively maintained project also but it does not seem to have a arm/64 port and I was not able to get it to work but it does have redshift listed in supported objects listing.

I have a follow up questions regarding having the ability to set up a lower AWS environment.

I just read up on terraform workspaces. Can workspaces be used to maintain DEV, STAGE, & PROD and have the environments be completely separate?

Also, is it typical to have multiple AWS accounts for separate environments?

@porion I am trying this myself, and finding the process to be very challenging.

I am running into “Cycle” errors, which seem to refer to circular references to stuff, but when I look up the things it refers to, they all seem to be legitimate records.

As far as whether it’s typical to have multiple AWS accounts for separate environments, the answer is “yes”. That follows the Well-Architected Framework model where using separate accounts is a good practice for separation of concerns.

1 Like

Workspaces in Terraform CLI refers to the ability to store several different named state files linked with the same directory of Terraform code, and swap between then. This is not sufficient on its own for multiple environments, because multiple environments typically require more things to be different than just the state file.

You might just need some different Terraform input variable settings, or environment variables - or you might even need different versions of your Terraform code in different environments.

You can use Terraform CLI workspaces as part of a multi-environment solution, but you need your own custom automation/wrappers to handle the rest.

Workspaces in Terraform Cloud (or Enterprise) are confusingly something different to workspaces in Terraform CLI. They supersede and replace the workspace concept in Terraform CLI, and do support associating different input/environment variables with a workspace, or using different Git repositories, directories, or branches with different workspaces.

I want to document my findings after 7 days of googling, experiments and discussion. Remember that the goal is to take an existing AWS production environment and copy/clone/reproduce it. Also understand that I am starting from basically zero knowledge.

  1. What I have learned thus far is that the task is non-trivial and even terraformer will not make it simple. period. There is no magic bullet.

  2. Tools like Terraform work best when used at the beginning of a project you start to use it as the tool for provisioning development, you then do development, and then use it as a tool to deploy Stage and Production. If on the other hand you have an existing infrastructure with a couple hundred resources (ec2, iam, s3, dynamodb, rds, lambdas, api_gateway endpoint) well this means you have some challenges.

  3. I believe that in order to reproduce something you have to first have an inventory. A As-is view of things that need to be reproduced and it must be complete. The way you could do this is to use the AWS CLI on one screen, and pull up a spreadsheet or code editor and start to inventory things manually e.g >aws dynamodb list-tables > table_list.txt . Now take the list of tables, go to AWS Admin UI and manually construct each table in the new Development AWS Account.

This is non-trivial due the mesh of objects including roles and groups.

  1. Terraformer not to be confused with Terraform, on the surface looks like a savior but has some limitations including: a. The version I have could not read/import redshift. Also, It dumps everything, even test objects that a developer created 2 years ago. Certainly we do not want this in the new environment.

  2. So regardless of whether you use Terraformer or not, you still have a lot of editing and desk checking before you can import objects into a new environment.

  3. Lastly, if you are using AWS. We learned you need to create a separate account that is not connected to your current running production account , In this account you need to a) add AWS Control Tower which will add 2 accounts, (Security & Logging), it will add an organization and make the account a management/payer account. b) you have to request that the original prod account join the newly created organization. c) after this you can create more accounts for development and stage. Again, non-trivial. I hope this helps someone.

That all sounds fairly accurate. I’d emphasise the amount of thought you need to apply - it isn’t just getting all the existing resources into basic code, but choosing what to include (as you mention some things just shouldn’t be there or are not part of the infrastructure), naming things in a sensible way, linking things together and choosing how to organise (modules, for_each, count, etc.).

Any tools/scripts that might exist or you create can only help some of the heavy lifting - there is still lots only a person who understands what is trying to be achieved can do.