Now that AWS has announced strong consistency for AWS I was thinking that there is no longer a need to use DynamoDB to manage locking. Is this something being considered? This would definitely simplify the bootstrapping of terraform state management. By reducing the dependency it would also free Terraform state management from potential DynamoDB outages.
The state file being strongly consistent doesn’t remove the need for state file locking. The reason for the lock is to prevent multiple parallel runs of Terraform. If you ran two copies of Terraform without locking at the same time you would end up either with errors (because you are trying to do a change that has already been applied) or multiple resources (both runs seeing a missing resource and creating it) which then can’t be saved in the state file - the run which finishes second would overwrite the first run data causing it to all be lost.
Or are you suggesting having locks within S3 itself?
I think the idea here is to implement the locking by writing a new object into the bucket whose presence represents the lock being held, and then to delete that object to represent the lock being released.
We’ve not yet done any research to see if S3’s new guarantees are sufficient for that model to be safe, and I also want to be transparent that this isn’t an area we’re actively working on: the existing S3/DynamoDB solution is working and meeting the use-case, so our efforts are naturally focused on other situations that are not yet solved. For that reason, we probably won’t be able to complete any such research in the very near future, but I’d personally like to include this simplification eventually if it’s safe to do so.
Indeed my suggestion is to add a lock file directly in s3 so that the prerequisites for using terraform can be simplified.
Dropping DynamoDB in favor of a lock file in S3 would take care of this issue mentioned in the documentation:
It is not possible to apply such fine-grained access control to the DynamoDB table used for locking, so it is possible for any user with Terraform access to lock any workspace state, even if they do not have access to read or write that state. If a malicious user has such access they could block attempts to use Terraform against some or all of your workspaces as long as locking is enabled in the backend configuration.
It does sound like an interesting option if the consistency guarantees work as suggested. But as @apparentlymart says that isn’t something HashiCorp are likely to work on in the near future, so your best bet would be to contribute such an option
Understood on the priority. I just wanted to make sure maintainers were aware of this new possibility.
@sodul FYI that part of the documentation you quoted is out-of-date - it’s now possible to control item-level access for DynamoDB in IAM.
I’d like to add the reference to the respective terraform GitHub feature request.
I understand that there are priorities however given the envisioned simplicity for the backend deployment itself, I’d appreciate getting rid of the DynamoDB setup.