Vault with consul backend error 500

I am building an infrastructure on AWS using pulumi. It is a cluster of 3 auto scaling groups, one for server nodes (3-3), one for client nodes (3-5) and one for vault (1-1) each node is an EC2 instance. My problem here is: I am facing some issues with installing vault which is I cant access the UI and the service is running but it cannot access consul for some reason and I have tried everything related to port 8200:

  1. Vault Service Running: The Vault service is running on the EC2 instance, as confirmed by the systemctl status vault command.
  2. Connection Issues: I was unable to connect to the Vault server using the Vault CLI (vault status) or access the Vault UI. The error message indicates a connection refusal, which suggests that either Vault is not listening on the expected address and port, or a network configuration is blocking the connection.
  3. Vault Configuration: The Vault configuration (vault.hcl) should be set up to listen on all interfaces ( and have TLS disabled for testing purposes. This configuration is necessary to access Vault from outside the EC2 instance.
  4. Network Configuration: The security group associated with your EC2 instance should allow inbound traffic on port 8200. There are no restrictive iptables rules on the instance itself.
  5. Storage Backend Warnings: The Vault logs show warnings related to storage migration checks with an Unexpected response code: 500. This indicates a potential issue with the communication between Vault and its storage backend (Consul in this case).
  6. Access Control: There are no ACLs.

My user data script for vault EC2s:

vault_user_data = """#!/bin/bash
set -e

# Install dependencies
sudo apt-get update && sudo apt-get install -y unzip curl jq awscli || { echo "Failed to install dependencies"; exit 1; }

# Install Consul
curl -fsSL${CONSUL_VERSION}/consul_${CONSUL_VERSION} -o || { echo "Failed to download Consul"; exit 1; }
unzip || { echo "Failed to unzip Consul"; exit 1; }
sudo mv consul /usr/bin/ || { echo "Failed to move Consul binary"; exit 1; }
rm || { echo "Failed to remove Consul zip file"; exit 1; }

if [ -z "$GOSSIP_KEY" ]; then
    # Generate a new gossip encryption key using Consul's keygen command
    GOSSIP_KEY=$(consul keygen)

sudo mkdir -p /etc/consul.d
# Create Consul server configuration
cat <<EOF | sudo tee /etc/consul.d/server.hcl
data_dir = "/opt/consul/data"
server = true
bootstrap_expect = 3
ui = true
client_addr = ""
    bind_addr = "{{ GetInterfaceIP \\"eth0\\" }}"
    advertise_addr = "{{ GetInterfaceIP \\"eth0\\" }}"
retry_join = ["provider=aws tag_key=Name tag_value=server"]
encrypt = "${GOSSIP_KEY}"

# Create systemd service file for Consul
cat <<EOF | sudo tee /etc/systemd/system/consul.service

ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d -server -ui
ExecReload=/bin/kill -HUP $MAINPID


# Start Consul
sudo systemctl daemon-reload
sudo systemctl enable --now consul || { echo "Failed to start Consul service"; exit 1; }

# Install Vault
curl -fsSL${VAULT_VERSION}/vault_${VAULT_VERSION} -o || { echo "Failed to download Vault"; exit 1; }
unzip || { echo "Failed to unzip Vault"; exit 1; }
sudo mv vault /usr/bin/ || { echo "Failed to move Vault binary"; exit 1; }
rm || { echo "Failed to remove Vault zip file"; exit 1; }

# Create a directory for Vault configuration
sudo mkdir -p /etc/vault.d /opt/vault/data

# Retrieve the instance's public IP address

# Create Vault configuration
cat <<EOF | sudo tee /etc/vault.d/vault.hcl
storage "consul" {
  address = ""
  path    = "vault/"
listener "tcp" {
  address     = ""
  tls_disable = "true"
ui = true

# Create systemd service file for Vault
cat <<EOF | sudo tee /etc/systemd/system/vault.service

ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill -HUP \$MAINPID


# Reload systemd, enable and start Vault service
sudo systemctl daemon-reload
sudo systemctl enable --now vault || { echo "Failed to start Vault service"; exit 1; }

# Wait for Vault to be ready
for i in {1..10}; do
    if curl -s --head | grep "200 OK"; then
        echo "Vault is up and running."
        echo "Waiting for Vault to start..."
        sleep 5

    if [ "$i" -eq 10 ]; then
        echo "Vault did not start in time."
        exit 1

# Initialize Vault
vault operator init -key-shares=1 -key-threshold=1 > /opt/vault/init.txt || { echo "Failed to initialize Vault"; exit 1; }

# Extract the unseal key and root token
UNSEAL_KEY=$(grep 'Unseal Key 1:' /opt/vault/init.txt | awk '{print $NF}')
ROOT_TOKEN=$(grep 'Initial Root Token:' /opt/vault/init.txt | awk '{print $NF}')

# Store the unseal key and root token in secure files
echo "${UNSEAL_KEY}" > /opt/vault/unseal_key.txt
echo "${ROOT_TOKEN}" > /opt/vault/root_token.txt

# Unseal Vault
vault operator unseal ${UNSEAL_KEY} || { echo "Failed to unseal Vault"; exit 1; }

# Store the root token in a file (for demonstration purposes)
echo "Root Token: ${ROOT_TOKEN}" > /opt/vault/root_token.txt

# Check if the gossip key exists in Vault
GOSSIP_KEY=$(curl -s --header "X-Vault-Token: ${ROOT_TOKEN}" http://${VAULT_IP}:8200/v1/secret/data/gossip | jq -r

# Store the gossip encryption key in Vault
curl --header "X-Vault-Token: ${ROOT_TOKEN}" --request POST --data "{\"data\": {\"key\": \"${GOSSIP_KEY}\"}}"

# Enable AppRole authentication
vault auth enable approle

# Create a role named 'nomad-consul-role'
vault write auth/approle/role/nomad-consul-role \
    token_policies="default" \
    secret_id_ttl=24h \
    token_ttl=1h \
    token_max_ttl=4h \

# Get RoleID and SecretID
ROLE_ID=$(vault read -field=role_id auth/approle/role/nomad-consul-role/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/nomad-consul-role/secret-id)

# Store RoleID and SecretID in Vault
vault kv put secret/nomad-consul/role-id value="$ROLE_ID"
vault kv put secret/nomad-consul/secret-id value="$SECRET_ID"