I am in the process of splitting a stack represented by one terraform state, into mulitple terraform states. Eg one state for network like vpc, gateway, etc; one state for EKS; one state for databases, etc. The databases tf config file use terrraform_remote_state data source, etc.
Notionally I think it would make sense for all the states associated with the same stack to be locked together. Eg if I make a change to the network tfstate in s3, there may be changes that affect the databases tfstate (like subnets) so I want to prevent the databases terraform config files from updating the databases tfstate while an update is happening in the network tfstate of the stack.
Does the remote state data source need to acquire the lock in order for the associated terraform apply to progress? If so then all good, but if not, it does not seem possible: the s3 backend lock that gets put in the dynamodb table is specific to the state file name in s3, it is not a key that I could choose and specify same key for multiple backends. So each tfstate I put in s3 has its own lock. There is no way of configuring multiple s3 backends to share a lock.
If I am correct that lock is per tfstate name so a dynamodb table can host mulitple locks (even for different stacks), seems like I could just create one bucket and one lock table for all states of all stacks. Then I would never face the conundrum of having to create a backend + table for a new stack or state.
Some questions:
- can I rely on data source to wait for the remote state to become unlocked?
- are there any gotchas that I should be aware of in using a mono bucket and mono lock table to host all locks and all tfstates?
Hi @schollii,
Terraform’s state locking is a per-configuration, per-workspace idea. Terraform has no built-in mechanism to control concurrency between configurations or between workspaces, and in particular the terraform_remote_state data source doesn’t acquire a lock because the locks are intended to prevent concurrent modifications, not concurrent reads. (This design assumes that all backends are able to write new state snapshots atomically, so that the latest will always appear valid to a concurrent reader.)
If you need cross-configuration concurrency limits then you’ll need to use something outside of Terraform to guarantee it. For example, in a former job (before I worked at HashiCorp on Terraform itself) we ran Terraform in an automation system that had its own sense of groups of jobs that would share a job queue so that we could limit the concurrency of all of them together by controlling how many workers were processing that queue.
OK good to know thanks. What is the locking strategy used by terraform, must be something like “first process to be able to create a lock object in dynamodb table has the lock; all terraform processes for a given state file path in s3 use the same lock object; retry lock creation every N seconds”.
I’m not too familiar with the S3 backend’s locking implementation myself, but indeed it does seem like it’s something like that. In the code I see it try to write an object into DynamoDB that contains a UUID and then the lock succeeded if it’s able to read back the same UUID.
You can see the details in the backend’s state client implementation if you’d like. The Lock and Unlock methods are the relevant parts for what we’re discussing here.
What about concurrent read and write (one process modifies state at same time as another reads it)? Isn’t there a chance of reader getting garbage (partially updated file), or a file which is still being changed eg if a terraform apply takes 10 minutes and the read happens in that period, it should really be blocking until the apply is done.
The snapshot writes by all backends should be atomic, so a reader will only ever see the entire previous snapshot or the entire new snapshot, never a partially-written new snapshot.
1 Like