Vault not able to sync data - Using ecs and s3 as storage backend

Vault
1

I have a HashiCorp Vault deployed in Amazon ECS with two tasks running. The setup includes AWS S3 as a backend storage for Vault and AWS KMS for auto-unsealing, with an Application Load Balancer (ALB) in front.

I’m encountering the following issues:

  1. Data Inconsistency: When accessing Vault through the ALB, auto-unsealing works intermittently. Creating secrets succeeds, but retrieving them shows inconsistency. Sometimes secrets are displayed, while other times they are not until multiple attempts.
  2. Data Persistence Issue: Only after restarting all ECS tasks can I consistently access secrets in Vault. Restarting is not an option when it comes to production…

Why am I experiencing these behaviors despite using the same S3 backend? How can I ensure data persistence and resolve the inconsistency in secret retrieval?

Any insights or suggestions would be greatly appreciated. Thank you!

2

Hello!

I am not super familiar with the S3 Storage Backend for Vault, but the documentation states that it does not support high availability. That may be causing the problems you are seeing. If you are wanting to run a multi-node Vault cluster for high availability purposes you will need to use a backend that supports it.

The S3 storage backend is used to persist Vault’s data in an Amazon S3 bucket.

  • No High Availability – the S3 storage backend does not support high availability.
  • Community Supported – the S3 storage backend is supported by the community. While it has undergone review by HashiCorp employees, they may not be as knowledgeable about the technology. If you encounter problems with them, you may be referred to the original author.