What are the correct status codes and behaviors in various state locking error scenarios?

Hi,

I’m trying to figure out the correct way to implement locking in a custom HTTP backend implementation, referring to the documentation at Backend Type: http | Terraform by HashiCorp and State: Locking | Terraform by HashiCorp.

  1. If a state lock request comes in to lock state “myState”, and the ID in the request matches the ID that has currently locked “myState”, is the correct response 200, LOCKED (423), or CONFLICT (409)?
  2. If a state lock request comes in to lock state “myState”, and the ID in the request does not match the ID that has currently locked “myState”, is the correct response LOCKED or CONFLICT?
  3. If a state saving request comes in, and the state is locked, and the ID supplied in the query parameter does not match the ID that has locked the state, what is the appropriate backend behavior?
  4. If a state saving request comes in, and the state is not locked, yet the client supplies a non-empty ID in a query parameter, what is the appropriate backend behavior?
  5. If a state saving request comes in, and the state is locked, and no ID is supplied in the query parameter, what is the appropriate backend behavior?

Thanks.

-mike

Hi @snowymike,

The http backend is quite liberal in what behaviors it will accept from servers because it is intended to be as easy as possible to integrate with pre-existing systems. I will answer your specific questions below, but you’ll see that the general theme here is that as much as possible you can choose a behavior that’s most appropriate for your situation.

  1. Terraform will not re-request the lock from the same client that’s already holding the lock, and so this situation should not occur. It’s typical to treat it the same as for question 2.
  2. In the event that the lock is already held, Terraform doesn’t not distinguish between the “Locked” or “Conflict” response codes. They are entirely equivalent and both mean that the request was successful but the lock is unavailable.
  3. In practical use Terraform will not perform any operations that would create new state snapshots if it does not successfully acquire a lock, so if this situation arises it would be an exceptional situation rather than expected behavior. If it’s practical for your implementation to verify lock IDs then it could be a useful addition for robustness against misbehavior, but Terraform itself considers the locking mechanism to be only advisory locks and so will avoid making requests that would depend on the server checking lock IDs. (Some implementations use the lock ID only for logging/analytics purposes, or don’t use the lock ID at all.)
  4. Again, this is not a situation that should arise in normal use of Terraform because Terraform will not try to create state snapshots without holding a lock. You can choose to block this for robustness if you wish. If you choose to block requests without an ID (in the next item) then it would be reasonable to treat incorrect lock IDs the same way, if your server is able to track the ID of the currently-held lock.
  5. This particular situation can arise if the backend is misconfigured to not include lock_address, since that will tell Terraform that it should not use locking at all. If you intend to always use locking then it would be appropriate to reject requests to create new state snapshots without including a lock ID. For requests to create new snapshots Terraform considers any non-successful status code to be equivalent and an error, so doesn’t practically matter which response code you use for this, but 400 Bad Request would be a reasonable general error response code to use for this situation, which will cause today’s Terraform to report the error HTTP error: 400.

Thanks for the wonderfully detailed reply. So it sounds like a lot of this is left up to the implementer to decide, and there isn’t a “any spec-compliant HTTP backend that supports locking must do it this way” type rule for the scenarios I described. With the exception of “if the lock is already held for a given state, return either LOCKED or CONFLICT (each equally acceptable) on a new state lock request”.