Vault Fails to init

Hi, i’m trying to setup vault, consul, nomad cluster combo, that is heavily based on:

This setup uses Packer to create an image that already has docker, docker-compose, vault, consul, nomad and consul-template installed and copies a few files for vault, consul, nomad, etc. into the image to be ran later.

Everything goes pretty smooth until initializaing vault:

digitalocean_droplet.server[0] (remote-exec): Error initializing: Put "http://127.0.0.1:8200/v1/sys/init": dial tcp 127.0.0.1:8200: connect: connection refused
digitalocean_droplet.server[0] (remote-exec): Unseal Key (will be hidden):

However i’m not sure why this is happening. Here is my vault-config.hcl:

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

api_addr = "http://127.0.0.1:8200"
ui       = true

Here’s the script that is ran to initialize vault:

#! /bin/bash

echo "Initialize Vault on server\n"

export VAULT_ADDR=http://127.0.0.1:8200

if [ $1 == "0" ]; then
  vault operator init -address=http://127.0.0.1:8200 > /root/startupOutput.txt
  vault operator unseal -address=http://127.0.0.1:8200 `grep "Unseal Key 1" /root/startupOutput.txt | cut -d' ' -f4`
  vault operator unseal -address=http://127.0.0.1:8200 `grep "Unseal Key 2" /root/startupOutput.txt | cut -d' ' -f4`
  vault operator unseal -address=http://127.0.0.1:8200 `grep "Unseal Key 3" /root/startupOutput.txt | cut -d' ' -f4`
fi

echo "Initialized Vault complete\n"
exit 0

My terraform setup runs it like so:

  provisioner "remote-exec" {
    inline = [
      "sleep 30",
      "export VAULT_ADDR=http://127.0.0.1:8200",
      "chmod +x /root/init_vault.sh",
      "/root/init_vault.sh ${count.index}",
    ]
  }

Can anyone help me figure out why Vault might be failing here?

Have you verified that consul is up and running before you start vault? It seems like vault isn’t initalizing to begin with, probably because it can’t connect to it’s configured storage.

The following commands were both run before running vault operator init

This is what I get when I run consul members:

digitalocean_droplet.server[0] (remote-exec): Node       Address          Status  Type    Build  Protocol  DC   Segment
digitalocean_droplet.server[0] (remote-exec): agent-s-1  10.116.0.2:8301  alive   server  1.9.1  2         dc1  <all>

This is the output when running consul info:

digitalocean_droplet.server[0] (remote-exec): agent:
digitalocean_droplet.server[0] (remote-exec):   check_monitors = 0
digitalocean_droplet.server[0] (remote-exec):   check_ttls = 0
digitalocean_droplet.server[0] (remote-exec):   checks = 0
digitalocean_droplet.server[0] (remote-exec):   services = 0
digitalocean_droplet.server[0] (remote-exec): build:
digitalocean_droplet.server[0] (remote-exec):   prerelease =
digitalocean_droplet.server[0] (remote-exec):   revision = ca5c3894
digitalocean_droplet.server[0] (remote-exec):   version = 1.9.1
digitalocean_droplet.server[0] (remote-exec): consul:
digitalocean_droplet.server[0] (remote-exec):   acl = disabled
digitalocean_droplet.server[0] (remote-exec):   bootstrap = false
digitalocean_droplet.server[0] (remote-exec):   known_datacenters = 1
digitalocean_droplet.server[0] (remote-exec):   leader = false
digitalocean_droplet.server[0] (remote-exec):   leader_addr =
digitalocean_droplet.server[0] (remote-exec):   server = true
digitalocean_droplet.server[0] (remote-exec): raft:
digitalocean_droplet.server[0] (remote-exec):   applied_index = 0
digitalocean_droplet.server[0] (remote-exec):   commit_index = 0
digitalocean_droplet.server[0] (remote-exec):   fsm_pending = 0
digitalocean_droplet.server[0] (remote-exec):   last_contact = never
digitalocean_droplet.server[0] (remote-exec):   last_log_index = 0
digitalocean_droplet.server[0] (remote-exec):   last_log_term = 0
digitalocean_droplet.server[0] (remote-exec):   last_snapshot_index = 0
digitalocean_droplet.server[0] (remote-exec):   last_snapshot_term = 0
digitalocean_droplet.server[0] (remote-exec):   latest_configuration = []
digitalocean_droplet.server[0] (remote-exec):   latest_configuration_index = 0
digitalocean_droplet.server[0] (remote-exec):   num_peers = 0
digitalocean_droplet.server[0] (remote-exec):   protocol_version = 3
digitalocean_droplet.server[0] (remote-exec):   protocol_version_max = 3
digitalocean_droplet.server[0] (remote-exec):   protocol_version_min = 0
digitalocean_droplet.server[0] (remote-exec):   snapshot_version_max = 1
digitalocean_droplet.server[0] (remote-exec):   snapshot_version_min = 0
digitalocean_droplet.server[0] (remote-exec):   state = Follower
digitalocean_droplet.server[0] (remote-exec):   term = 0
digitalocean_droplet.server[0] (remote-exec): runtime:
digitalocean_droplet.server[0] (remote-exec):   arch = amd64
digitalocean_droplet.server[0] (remote-exec):   cpu_count = 1
digitalocean_droplet.server[0] (remote-exec):   goroutines = 80
digitalocean_droplet.server[0] (remote-exec):   max_procs = 1
digitalocean_droplet.server[0] (remote-exec):   os = linux
digitalocean_droplet.server[0] (remote-exec):   version = go1.15.6
digitalocean_droplet.server[0] (remote-exec): serf_lan:
digitalocean_droplet.server[0] (remote-exec):   coordinate_resets = 0
digitalocean_droplet.server[0] (remote-exec):   encrypted = false
digitalocean_droplet.server[0] (remote-exec):   event_queue = 0
digitalocean_droplet.server[0] (remote-exec):   event_time = 1
digitalocean_droplet.server[0] (remote-exec):   failed = 0
digitalocean_droplet.server[0] (remote-exec):   health_score = 0
digitalocean_droplet.server[0] (remote-exec):   intent_queue = 1
digitalocean_droplet.server[0] (remote-exec):   left = 0
digitalocean_droplet.server[0] (remote-exec):   member_time = 2
digitalocean_droplet.server[0] (remote-exec):   members = 1
digitalocean_droplet.server[0] (remote-exec):   query_queue = 0
digitalocean_droplet.server[0] (remote-exec):   query_time = 1
digitalocean_droplet.server[0] (remote-exec): serf_wan:
digitalocean_droplet.server[0] (remote-exec):   coordinate_resets = 0
digitalocean_droplet.server[0] (remote-exec):   encrypted = false
digitalocean_droplet.server[0] (remote-exec):   event_queue = 0
digitalocean_droplet.server[0] (remote-exec):   event_time = 1
digitalocean_droplet.server[0] (remote-exec):   failed = 0
digitalocean_droplet.server[0] (remote-exec):   health_score = 0
digitalocean_droplet.server[0] (remote-exec):   intent_queue = 0
digitalocean_droplet.server[0] (remote-exec):   left = 0
digitalocean_droplet.server[0] (remote-exec):   member_time = 1
digitalocean_droplet.server[0] (remote-exec):   members = 1
digitalocean_droplet.server[0] (remote-exec):   query_queue = 0
digitalocean_droplet.server[0] (remote-exec):   query_time = 1

You don’t have a consul quram.

 leader = false

You need your cluster to be available. Are you starting only a single node? That’s the issue.

1 Like

So i’m deploying a Consul and Nomad clusters, and vault on each of those servers where consul is running the server agent. The error happens when starting to create the first Consul server agent and first vault agent

It might be easier to show you how my setup is configured:

It varies from the one I based it off of, by removing some of the firewall and domain name setup.

Also, any feedback, suggestions, comments, are welcome. I’m very new to this and I’m keen on learning what i’m doing wrong, bad, etc.