I am using terraform open source for creating and managing GCP resources. While performing terraform apply, I have issued control+c twice and it has corrupted the state-file as per the given warnings.
I wanted to know the possible ways to recover from this.
Lets say I have 5 rss defined in TF file. and while 5th rss is being created if I issue control+c twice. Terraform apply will exit immediately and state file will not have that rss but it will be created in GCP project.
Subsequent apply commands are waiting for a long time(30min) to report that the rss already exist. and destroy will not touch the rss as it is not in the state-file.
The only possible option seems to be deleting the rss manually or import it to the state file via terraform import command. I want to handle this scenario in an automated fashion.
You might say dont issue control+c while terraform apply, but there may be situations like:
- What if I want to stop the execution in middle for some reason?
- If am running terraform from a GCP compute engine, and it crashes which is more or less like control+c interrupts. I have tested the scenario.
- If am running terraform on a cloudbuild, and wanted to stop it for some reason?
I tried with both local and remote state-files:
local: State-file data loss happened, refresh is not getting the orphaned rss which was created as part of initial terraform apply which was interrupted twice. terraform apply for the second run hangs on creating the same rss and after 30~40 mins it says it already exists.
Remote: Initial cancelled terraform run holds the lock on the state-file, subsequent runs have to be run only by passing -lock=false option(which is not recommended), which immediately says all rss exists.
so how can we handle these scenarios? If terraform can’t handle it can we say its tool reliability issue or how to address?
In these sort of situations you need to manually fix the state file and remove the lock, once you have investigated the situation.
It isn’t safe to just ignore the lock as you would no longer be preventing multiple simultaneous Terraform apply instances. Instead you need to check that there are no other instances of Terraform running and then use the force unlock command.
If you interrupt an apply as you say the state file might now be incorrect. In this case you will need to check what is needed and may need to import resources, remove things from state or remove resources using other systems (eg a cloud cli or Web interface).
All of these require investigation and intelligent decisions, so would in general be difficult / unsafe to try to automate.
@stuart-c certainly gave the correct answer to your question. I just want to add that some of the cleanup steps, the obvious ones like unlocking the state, checking state consistency, and reporting vital information useful to the human to remedy the situation via imports and state rm, can indeed be automated.
Sometimes anti-patterns, such as ungracefully terminating running terraform processes, must be followed to workaround the side effects of larger anti-patterns, such as waiting around for dozens of minutes for terraform operations to complete due to a poorly-architected project or when troubleshooting the numerous bugs in terraform modules…
Indeed I have found my left pinky and index finger on Ctrl and C, respectively, and having to follow up with a terraform force-unlock -force <lock-id>, so frequently that I decided to implement a SIGINT handler in my scripts to handle these cleanup steps for me. It’d be great if I could automate the hard part too, but I have never had a problem with corrupt state needing to be fixed, so for me it’s kind of a pre-mature optimization of the 1% case.