Hello,
I am having a Terraform project set up done by earlier team. They initially created the required AWS resources including EKS clusters, self-managed nodes etc through Terraform configs.
But later for some reasons, we discontinued using Terraform and made lots of changes directly through AWS console/API. Majorly, we have switched from self-managed nodes to EKS managed node groups, added some new EC2 servers for various purposes etc.
I want an expert advice on this scenario here. Is it possible to re-sync the new infrastructural changes back to existing Terraform project? Or should I go with defining a whole infrastructure again by discontinuing the earlier set up?
Thanks in advance.
Not an impossible task from what I’ve seen.
But you need to sync i.e update every parameter that has been changed manually into your existing TF codebase, it depends on how many things have been changed.
Do a terraform refresh to sync the changes done manually into your state file and a plan to see the behavior.
Take backup before doing any major step.
If TF codebase is very old, I suggest a from scratch, but it depends on your scenario.
1 Like
It sounds like you aren’t sure what might have changed in the meantime since Terraform was last used, and that presents a problem:
Terraform only tracks in its state the objects that it created, or that it was explicitly asked to “adopt” using the import features. If anyone has created anything entirely new since the last use of Terraform then Terraform will have no awareness of it at all, and so won’t know to track it unless you explicitly import it.
You mentioned switching wholesale from directly-managed nodes to EKS managed node groups, which I guess means that your most recent Terraform state will be tracking a bunch of individual EC2 instances, or an autoscaling group, or similar, and crucially it won’t have any knowledge about your newer EKS node groups.
With that all considered, I would guess that trying to describe the current system that exists using a fresh Terraform configuration and then importing your existing infrastructure into it would probably be the shortest and least fraught path here, because:
- you can presumably inspect the current situation in the running system to reverse-engineer any details you aren’t sure about, whereas the old system that you existing Terraform configuration was written for essentially no longer exists and so you’d need to rely on institutional memory to understand how it was designed and why.
- starting from a fresh Terraform state and gradually importing things into it means you can approach this task gradually with less risk to the running system; you can hopefully adopt one layer into Terraform at a time, reach the point where it plans “no changes” (meaning the desired state and actual state are converged), and then move on to the next layer. (I’m using “layer” in a general sense here; you will need to decide what makes sense as cut points based on how the system is designed, but the idea would be to at each point have as clear as possible a line between what is currently Terraform-managed and what isn’t, so you can communicate with the rest of your team and make sure coworkers aren’t effectively undoing your work by making even more changes outside of Terraform.)
- You presumably now have the experience of running this real system and know what sorts of changes are likely to happen routinely and what sorts of changes are less common. That is likely to influence how you design your Terraform configuration, and trying to bring forward design decisions from the earlier phase of the system may force you down a less useful design path as you try to keep the old Terraform state matching when you update things.
Without seeing the details of what you have right now I can’t be certain about this suggestion. If some of the assumptions I’ve stated don’t seem to be true then my advice might not be valid, but this is my best attempt based on what I have understood.
1 Like
Thank you for your helpful reply. Yes, the Terraform code is too old and I am seeing good amount of deprecation errors when I try to run terraform plan or refresh. So I was thinking on the same line as you suggested, to start from scratch and match the running infrastructure into new codebase. May be I can try that with some other region so my current live applications won’t affect.
Thank you for your thoroughand valuable reply. I am quite new to this Terraform project, but as much as I know, there are many new changes made to the infrastructure after it got initially created by earlier team. I agree with your thoughts here about going with fresh Terraform configuration and then importing your existing infrastructure into it.