Cassandra storage backend timeouts

We are experiencing intermittent periods where we will get a considerable number of requests that fail with this error:

“1 error occurred: * gocql: no response received from cassandra within timeout period”

When looking through the documentation for the Cassandra storage backend I see that the connetion_timeout value is available.
There does appear to be an issue where the default is set to 0s that is in PR. We can set that manually and will be trying that. However, I do not think that our issue is related to connect timeout but is rather a request_timeout

Reading through the gocql documentation there appears to be two timeout defaults:
Timeout time.Duration // connection timeout (default: 600ms) ConnectTimeout time.Duration // initial connection timeout, used during initial dial to server (default: 600ms)

Though it seems like if you don’t provide a config that vault literally defaults to a 0s connect timeout.

My questions: Does vault provide overriding a request timeout (similar to cqlsh) or does it only use the one connect timeout? Is the connection_timeout literally our issue and we’re falling victim to the issue linked above?

Any discussion/insight would be helpful. Thanks

My guess is that 0s timeout is either undefined or gocql ignores it as bogus and uses 600ms. So we’re likely working with 600ms on both in the underlying gocql. Just looking for some more fine grained details on the vault level config and how it might affect operation as it appears that our Cassandra cluster is under no particularly noticeable load.

I face exactly this issue

2022-03-31T12:05:07.946+0200 [INFO]  core: restoring leases
[ERROR] expiration: error restoring leases: error="failed to scan for leases: list failed at path \"\": gocql: no response received from cassandra within timeout period"

is any one know how to set request timeout in cassandra back end ?