We are creating our entire AWS infrastructure using terraform. Its creating around 500 resources. We are using gitlab pipeline to run a script which does “terraform apply” and it starts creating required AWS resources. In some cases gitlab runner dies in the middle of the run and the script ends abruptly. There is another case as well where ECS service deployments gets stuck and the gitlab pipeline timeouts out after certain time and that also ends the run abruptly. These and few other cases where the pipeline which run the script ends abruptly and leave the terraform state without getting updated with resources it already created. When I run the pipeline again to continue applying from where it left, it gives error that resources already exists and the pipeline fails. The error shows many resources which are already created but not managed by the TF state for which I was running the pipeline. If it would have been one or two resources, I can do terraform import and import those resources into the state. But since there are many resources, its getting difficult to add them all to state and get the state to latest.
The help which I am looking for is:
- How can this situation be handled where the the terraform apply is interrupted in the middle and state not updated properly?
- Has anyone come across any code/script which can read through the errors in pipeline about already existing resource and add them to state to recover the state of the infrastructure ?
Please let me know if you need any other details.