S3 backend state lock won't release

Terraform
1

I am using s3 as my backend, with a dynamoDB table for state locking. When I try to run a plan, I am getting a message that a previous plan I ran but did not complete is holding the state lock. I try to force unlock and get “Local state cannot be unlocked by another process.”

I am not using a local state - the path in the lock message clearly shows my s3 backend. I have no local terraform processes running to kill - to make sure, I even restarted my dev computer. How can I release this lock?

2

terraform force-unlock -force 4XX-XXXXXX-XXd

Failed to unlock state: LocalState not locked

terraform force-unlock 4XX-XXXXXX-XXd

Local state cannot be unlocked by another process

3

The last ditch way is to simply delete the lock object in the dynamodb table. It should be literally the only thing in the table. You can do this through the amazon web ui, or cli (but much easier in the web ui).

4

##Solution

I have a quite complex environment, and I have no console access to this account. The LockID field in the dynamoDB table was not what I expected. I could not find these specific instructions anywhere, so I thought I would describe how to actually do this using the CLI for anyone else who may get stuck with state lock hangups.

I viewed the lock table in in my config first, then I verified it existed in the AWS account I was currently using. The profile is my named AWS CLI profile for this project. If you only have one AWS CLI account configured, you can leave off the --profile=<your_profile> command from all of these and probably skip this validation step.

aws dynamodb list-tables --profile=<your_profile>

Then I listed all items in the table. I had quite a few states in my table because of my complex env.

aws dynamodb scan --table-name <your_table> --profile=<your_profile>

The LockID for deleting the state is not the lockID shown in the output telling you that your state was locked. My LockID, which is what I needed to put in the command below, looked like a path to a terraform.tfstate file. What the error message showed me as the LockID is a different field, just called ID. Currently locked states will have an Info attribute that has that ID so you can confirm you are targeting the correct state. To verify that more easily:

aws dynamodb get-item \
--table-name <your_lock_table> \
--key '{
      "LockID": {"S":"<lock_id_discovered_during_previous_step>"}
  }' \
--profile=<your_profile>

Once you are totally sure you are targeting the right state, removing the entry cleared the lock.

aws dynamodb delete-item \
--table-name <your_lock_table> \
--key '{
      "LockID": {"S":"<lock_id_discovered_during_previous_step>"}
  }' \
--profile=<your_profile>
1 Like