Vault aws secret engine - context deadline exceeded

Vault
1

Hi there,

We’ve installed Vault V1.5.5 with Consul V1.8.4 as backend. We have also installed the aws secret engine.
Very often, in fact 1 in 3 queries of vault to generate a temporary IAM user and key, result in a timeout or ‘context deadline exceeded error’. We and AWS support through cloudtrail has verified that AWS API throttling is not the cause of error.

Through CLI we end up seeing error like this:

vault read voltron-cli/aws/us-east-1/agile/vpc/cta-platform-build-vpc-9e56a02d-5280-48ce-89f6-5dc1608501b2/creds/vpc-9e56a02-write
Error reading voltron-cli/aws/us-east-1/agile/vpc/cta-platform-build-vpc-9e56a02d-5280-48ce-89f6-5dc1608501b2/creds/vpc-9e56a02-write: context deadline exceeded

We’ve checked the vault and consul telemetry data and nothing pops up. has anyone seen anything similar ?

Thanks,

2

Turn on debug logs - do they show anything during this event?
Is it consistently 1 in 3 queries?
Is Vault behind a LB?
Is Consul deployed based on the reference architecture?
How long does the context deadline exceeded message take to appear?