I ran a terraform apply from a clean project for some resources in GCP.
Mid run the script encountered an error, probably a misconfiguration on my part.
Error printed:
Error: Error applying IAM policy for service account ‘service-account’: Error setting IAM policy for service account ‘service-account’: googleapi: Error 400: Identity Pool does not exist (project-id.svc.id.goog). Please check that you specified a valid resource name as returned in the name attribute in the configuration API., badRequest
But now to the problem, if I try to run an apply again terraform seems to have created many resources that was never saved into the state. Such as artifact registries, service accounts, pub/sub topics etc.
Is this a bug or am I doing something wrong here? How can I recover from this broken state? Is the only way forward to manually go in and delete everything?
Given that you seem to have a number of resources which are not tracked in the state, it sounds like you terminated the Terraform process before it was able to save all the changes that were in progress. If that’s not the case, then it would indicate a bug in the provider, however unless the provider completely crashed, I would not expect multiple resources to be lost.
As far as recovery, the options are either to manually remove the resources directly within GCP, or to import the resources into Terraform. Both of these are mostly manual operations, so it’s up to you which one is easier for your particular situation.
I think I figured out what I’m doing wrong, I ran the apply in my CI but did not properly configure the remote state, hence I think a local state was created and then lost.
But good to know that it likely is a provider issue if a single resource is not saved (I have experienced that in the past with cloud functions in GCP).
A specific kind of provider bug that can cause the symptom of one object not being tracked arises when a provider is wrapping a series of separate API calls but fails partway through.
In that case a provider is supposed to return an error along with a state object describing whatever was already changed, but sometimes providers incorrectly return only the error in that case, which tells Terraform Core that it should retain whatever was the prior state of the object. If the operation was to create the object then it has no prior state at all, and so Terraform incorrectly believes the object doesn’t exist.
Although in this case that clearly wasn’t the cause, if you do encounter situations where only one object is missing after its creation failed then it’s worth reporting that as a bug in the provider’s source repository.