Vault with consul backend error 500

Vault
1

I am building an infrastructure on AWS using pulumi. It is a cluster of 3 auto scaling groups, one for server nodes (3-3), one for client nodes (3-5) and one for vault (1-1) each node is an EC2 instance. My problem here is: I am facing some issues with installing vault which is I cant access the UI and the service is running but it cannot access consul for some reason and I have tried everything related to port 8200:

  1. Vault Service Running: The Vault service is running on the EC2 instance, as confirmed by the systemctl status vault command.
  2. Connection Issues: I was unable to connect to the Vault server using the Vault CLI (vault status) or access the Vault UI. The error message indicates a connection refusal, which suggests that either Vault is not listening on the expected address and port, or a network configuration is blocking the connection.
  3. Vault Configuration: The Vault configuration (vault.hcl) should be set up to listen on all interfaces (0.0.0.0:8200) and have TLS disabled for testing purposes. This configuration is necessary to access Vault from outside the EC2 instance.
  4. Network Configuration: The security group associated with your EC2 instance should allow inbound traffic on port 8200. There are no restrictive iptables rules on the instance itself.
  5. Storage Backend Warnings: The Vault logs show warnings related to storage migration checks with an Unexpected response code: 500. This indicates a potential issue with the communication between Vault and its storage backend (Consul in this case).
  6. Access Control: There are no ACLs.

My user data script for vault EC2s:

vault_user_data = """#!/bin/bash
set -e

# Install dependencies
sudo apt-get update && sudo apt-get install -y unzip curl jq awscli || { echo "Failed to install dependencies"; exit 1; }

# Install Consul
CONSUL_VERSION="1.11.1"
curl -fsSL https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip -o consul.zip || { echo "Failed to download Consul"; exit 1; }
unzip consul.zip || { echo "Failed to unzip Consul"; exit 1; }
sudo mv consul /usr/bin/ || { echo "Failed to move Consul binary"; exit 1; }
rm consul.zip || { echo "Failed to remove Consul zip file"; exit 1; }

if [ -z "$GOSSIP_KEY" ]; then
    # Generate a new gossip encryption key using Consul's keygen command
    GOSSIP_KEY=$(consul keygen)
fi

sudo mkdir -p /etc/consul.d
# Create Consul server configuration
cat <<EOF | sudo tee /etc/consul.d/server.hcl
data_dir = "/opt/consul/data"
server = true
bootstrap_expect = 3
ui = true
client_addr = "0.0.0.0"
    bind_addr = "{{ GetInterfaceIP \\"eth0\\" }}"
    advertise_addr = "{{ GetInterfaceIP \\"eth0\\" }}"
retry_join = ["provider=aws tag_key=Name tag_value=server"]
encrypt = "${GOSSIP_KEY}"
EOF

# Create systemd service file for Consul
cat <<EOF | sudo tee /etc/systemd/system/consul.service
[Unit]
Description=Consul
Documentation=https://www.consul.io/docs/
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d -server -ui
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

# Start Consul
sudo systemctl daemon-reload
sudo systemctl enable --now consul || { echo "Failed to start Consul service"; exit 1; }

# Install Vault
VAULT_VERSION="1.9.0"
curl -fsSL https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip -o vault.zip || { echo "Failed to download Vault"; exit 1; }
unzip vault.zip || { echo "Failed to unzip Vault"; exit 1; }
sudo mv vault /usr/bin/ || { echo "Failed to move Vault binary"; exit 1; }
rm vault.zip || { echo "Failed to remove Vault zip file"; exit 1; }

# Create a directory for Vault configuration
sudo mkdir -p /etc/vault.d /opt/vault/data

# Retrieve the instance's public IP address
PUBLIC_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)

# Create Vault configuration
cat <<EOF | sudo tee /etc/vault.d/vault.hcl
storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}
ui = true
EOF

# Create systemd service file for Vault
cat <<EOF | sudo tee /etc/systemd/system/vault.service
[Unit]
Description=Vault
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill -HUP \$MAINPID
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd, enable and start Vault service
sudo systemctl daemon-reload
sudo systemctl enable --now vault || { echo "Failed to start Vault service"; exit 1; }


# Wait for Vault to be ready
for i in {1..10}; do
    if curl -s --head http://127.0.0.1:8200/v1/sys/health | grep "200 OK"; then
        echo "Vault is up and running."
        break
    else
        echo "Waiting for Vault to start..."
        sleep 5
    fi

    if [ "$i" -eq 10 ]; then
        echo "Vault did not start in time."
        exit 1
    fi
done

# Initialize Vault
vault operator init -key-shares=1 -key-threshold=1 > /opt/vault/init.txt || { echo "Failed to initialize Vault"; exit 1; }

# Extract the unseal key and root token
UNSEAL_KEY=$(grep 'Unseal Key 1:' /opt/vault/init.txt | awk '{print $NF}')
ROOT_TOKEN=$(grep 'Initial Root Token:' /opt/vault/init.txt | awk '{print $NF}')

# Store the unseal key and root token in secure files
echo "${UNSEAL_KEY}" > /opt/vault/unseal_key.txt
echo "${ROOT_TOKEN}" > /opt/vault/root_token.txt

# Unseal Vault
vault operator unseal ${UNSEAL_KEY} || { echo "Failed to unseal Vault"; exit 1; }

# Store the root token in a file (for demonstration purposes)
echo "Root Token: ${ROOT_TOKEN}" > /opt/vault/root_token.txt

# Check if the gossip key exists in Vault
GOSSIP_KEY=$(curl -s --header "X-Vault-Token: ${ROOT_TOKEN}" http://${VAULT_IP}:8200/v1/secret/data/gossip | jq -r .data.data.key)


# Store the gossip encryption key in Vault
curl --header "X-Vault-Token: ${ROOT_TOKEN}" --request POST --data "{\"data\": {\"key\": \"${GOSSIP_KEY}\"}}" http://127.0.0.1:8200/v1/secret/data/gossip


# Enable AppRole authentication
vault auth enable approle



# Create a role named 'nomad-consul-role'
vault write auth/approle/role/nomad-consul-role \
    token_policies="default" \
    secret_id_ttl=24h \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_num_uses=0

# Get RoleID and SecretID
ROLE_ID=$(vault read -field=role_id auth/approle/role/nomad-consul-role/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/nomad-consul-role/secret-id)

# Store RoleID and SecretID in Vault
vault kv put secret/nomad-consul/role-id value="$ROLE_ID"
vault kv put secret/nomad-consul/secret-id value="$SECRET_ID"
"""