Vault DynamoDB Multi-Region

Vault
1

Hello there,

We are trying to deploy Vault Community Edition to be multi-region and use DynamoDB as a backend for this purpose.

And unfortunately, I am experiencing issues with my Vault application that uses DynamoDB tables across two AWS regions. Below are the details of my setup and the specific issues I’m encountering:
Setup Details:
Regions: us-east-2 and us-west-2
DynamoDB Tables: Created empty DynamoDB tables in both regions and enabled multi-region replication.
Streams: stream_enabled = true
stream_view_type = “NEW_AND_OLD_IMAGES”

Application Initialization: Started the Vault application in us-east-2 first, which populated the DynamoDB table with default settings and information.(no mountpoint or secrets added) Subsequently started the Vault application in us-west-2, also with default settings. The application is running in Kubernetes.

Issues Observed: The Vault application running in us-west-2 is facing problems. The following errors were taken from the pods in us-west-2:
2024-06-07T13:02:31.489Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[“req_fw_sb-act_v1”] 2024-06-07T13:02:31.490Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error=“unsupported protocol” 2024-06-07T13:22:10.074Z [DEBUG] core.cluster-listener: creating rpc dialer: address=vault-1.vault-internal:8201 alpn=req_fw_sb-act_v1 host=fw-4c877c26-eb06-6f61-5d35-82e1efe6530f 2024-06-07T13:22:09.073Z [DEBUG] core: forwarding: error sending echo request to active node: error=“rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: remote error: tls: internal error"”

When trying to curl hostname we get either:

    • local node not active but active cluster node not found
  2. 503 Service Unavailable

Questions:

  1. Could starting the Vault application with default settings in both regions independently cause any conflicts or overwrites in the DynamoDB tables? How should the initialization process be handled to avoid such issues?
  2. Are there any specific configuration settings or best practices for using Vault with DynamoDB in a multi-region setup? Could there be Vault-specific settings that need to be adjusted for optimal performance?
  3. Are there any additional best practices for using DynamoDB in a multi-region setup with applications like Vault Community Edition to ensure high availability and performance?

This is the documentation we used to implement Dynamodb as backend: DynamoDB - Storage Backends - Configuration | Vault | HashiCorp Developer

Any input would be helpful