I apologize if this has been asked before or if I missed it in the documentation. I’m trying to put out a small fire at work and need a quick response from an expert.
We have multiple processes that are trying to obtain a “lock”. We implemented the lock using Consul by having those processes try to store the same key in the database with the “cas” option set to 0. I think the assumption was that the first process would be able to store the key. The second process would see that the key already exists and return a false status indicating that the key was not stored.
Underneath the hood, is Consul providing the first process a mutual exclusive lock on the key? I ask because at work our application might be behaving like multiple processes are getting the lock on the resource. i.e both are able to store a key in Consul. Additionally, trying to investigate this issue I came across a document that describes how to implement a locking mechanism using Consul’s session API:
We are not using the Session API. We simply store a key using the cas option set to 0.
That leads to me believe that the implemeters of this locking mechanism may have made some assumptions about what Consul is doing under the hood. I do know that setting cas=0 does provide some degree of locking. it’s been working fine for a number of years and we can see processes waiting for locks to be released. However, we added a resource over the weekend that has a much greater parallel demand of locks than other resources.
It’s possible that the locking is fine and another area of our code is broken. I’m hoping someone here can help confirm whether our locking srategy is valid or not
or describe why the session api was documented practice for locking.
BTW,I think we are using Consul 1.0.0 on Redhat 6.2