We have a 3 node/3 AZ load-balanced ec2 cluster running vault 1.17.5, using dynamodb as a backend. The load on the service is relatively light, primarily getting hit by chef client with authentication (mix of iam and approle) and kv2 read requests from a few hundred nodes. Chef client only runs every 15 minutes, so it’s not a situation of everyone shouting at once.
The vault service itself seems to be running fine, and nothing in the server logs is amiss. However, we recently happened across a dynamodb metric “ConditionalCheckFailedRequests” that seems to suggest all may not be well.
Dynamodb metrics say successful reads (mix of GetItem (200-400 requests over 3 hours) and Query (0-200 over 3 hours)) and writes (BatchWriteItem (15-200 req)) are reasonable levels. However, the ConditionalCheckFailedRequests count is staying right around 120 for the last 3 hours, ~600 for the last 24h.
It’s not clear if this is a problem, or if this is a design feature of vault. Can anyone shed more light on what this is in terms of vault? If it is an issue, where should I go digging to figure out how to resolve it?
thanks