Vault TLS Handshake Error

Hi guys.

I downloaded Vault’s executable binary file and installed Vault on my IOT device. I produced a self-signed certificate with Openssl:

$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out vault.crt -keyout vault.key

I also wrote a bash script for the unseal process:
vault_seal_manager.sh

#!/bin/sh
export vault=/usr/bin/vault
vault_unseal="$vault operator unseal"
vault_status="$vault status"
vault_read="$vault read"
vault_init="$vault operator init"
vault_login="$vault login"
vault_kv_enable="vault secrets enable -version=1 kv" vault_init_status=($vault_status | awk '/Initialized[[:space:]]/ {print $2} ')

function check_unsealed(){
vault_status &> /dev/null if [[ ! ? == “0” ]]
then
echo "VaultSealManager-[$(date +’%X %x’)]-[ERROR]: Local Vault instance was unsuccessfully unse
exit 1
fi
}

function get_keys(){
vault_key_1=(sed '1!d' /etc/vault.d/keys_token.txt) # (vault_read -field=value secret/vault/keys vault_key_2=(sed ‘2!d’ /etc/vault.d/keys_token.txt) # $(vault_read -field=value secret/vault/keys vault_key_3=(sed ‘3!d’ /etc/vault.d/keys_token.txt) # $(vault_read -field=value secret/vault/keys vault_key_4=(sed ‘4!d’ /etc/vault.d/keys_token.txt) # $(vault_read -field=value secret/vault/keys vault_key_5=(sed ‘5!d’ /etc/vault.d/keys_token.txt) # $(vault_read -field=value secret/vault/keys vault_root_token=(sed ‘7!d’ /etc/vault.d/keys_token.txt)
if [[ -z “$vault_key_1” ]] || [[ -z “$vault_key_2” ]] || [[ -z “$vault_key_3” ]] || [[ -z "vault_k then echo "VaultSealManager-[(date +’%X %x’)]-[ERROR]: Error retrieving unseal keys from Vault secr
exit 1
fi
}

function unseal_vault(){
$vault_unseal vault_key_1 &> /dev/null; status_1=?
$vault_unseal vault_key_2 &> /dev/null; status_2=?
$vault_unseal vault_key_3 &> /dev/null; status_3=?
# Only need three to unseal
#$vault_unseal vault_key_4 &> /dev/null; #status_4=?
#$vault_unseal vault_key_5 &> /dev/null; #status_5=?

if [[ ! $status_1 == "0" ]] || [[ ! $status_2 == "0" ]] || [[ ! $status_3 == "0" ]]  #  || [[ ! $st
then
    echo "VaultSealManager-[$(date +'%X %x')]-[ERROR]: Error unsealing local Vault instance!"
    exit 1
fi

}

function set_kv(){
$vault_kv_enable &> /dev/null;
}

function vault_initialize() {
$vault_init > /etc/vault.d/new_keys.txt

    cat > /etc/vault.d/keys_token.txt << EOF
    $( awk '{print $4}' /etc/vault.d/new_keys.txt)

EOF
}

function vault_login() {
$vault_login $vault_root_token &> /dev/null;
}

function main(){
vault_status &> /dev/null if [[ ? == “0” ]]
then
echo “VaultSealManager-[$(date +’%X %x’)]-[IFNO]: Local Vault instance is already unsealed!”
exit 0
fi

    if [ "${vault_init_status}" = "false" ]; then
          vault_initialize

    fi
    rm -rf /usr/bin/new_keys.txt &> /dev/null

get_keys
echo "VaultSealManager-[$(date +'%X %x')]-[INFO]: Got unseal keys successfull..."
echo "VaultSealManager-[$(date +'%X %x')]-[INFO]: Attempting to unseal local Vault instance with ac

unseal_vault
echo "VaultSealManager-[$(date +'%X %x')]-[INFO]: Checking local seal status..."

vault_login

check_unsealed
echo "VaultSealManager-[$(date +'%X %x')]-[INFO]: Local Vault instance is now unsealed!"

set_kv

}

main
exit 0

Also, my .hcl file that I configure Vault for filesystem is below:

vault.hcl

storage “file” {

path = “mnt/vault/data/test2”

}

listener “tcp” {

address = “127.0.0.1:8200”

tls_cert_file = “/etc/vault.d/vault.crt”

tls_key_file = “/etc/vault.d/vault.key”

}

disable_mlock = true

After making the necessary settings, I created the vault.service configuration file in the /etc/systemd/system/multi-user.target.wants directory to stand up Vault from systemd:

vault.service:

[Unit]

Description=vault server

[Service]

Restart=on-failure

ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl

ExecStop=/usr/bin/vault operator step-down

[Install]

WantedBy=multi-user.target

In addition, I added vault-seal-process.service to the same directory (/ etc / systemd / system) to stand up the bash script I created for unseal process via systemd:

vault-seal-manager.sh

[Unit]

Description=Vault Initialization and Seal Manager

After=vault.service

[Service]

User=root

ExecStart=/usr/bin/vault_seal_manager.sh

[Install]

WantedBy=multi-user.target

After making all these settings, I booted the system and waited for Vault to stand up. But in this case he gives the error that he normally does not give.
My problem is this: When I run the bash script from the command line, it initializes Vault without any error and unlocks the seal:

$ vault_seal_manager.sh

VaultSealManager-[09:29:08 AM 02/10/2020]-[INFO]: Got unseal keys successfull…

VaultSealManager-[09:29:08 AM 02/10/2020]-[INFO]: Attempting to unseal local Vault instance with acquired unseal keys…

VaultSealManager-[09:29:09 AM 02/10/2020]-[INFO]: Checking local seal status…

VaultSealManager-[09:29:09 AM 02/10/2020]-[INFO]: Local Vault instance is now unsealed!

If I run Vault through systemd and run my bash script from the command line, there is no problem. But when I run the bash script via systemd, I get an error like the following:

root@ubuntu:/home/robust2# systemctl status vault-seal-manager.service

● vault-seal-manager.service - Vault Initialization and Seal Manager

Loaded: loaded (/etc/systemd/system/vault-seal-manager.service; enabled; vendor preset: enabled)

Active: failed (Result: exit-code) since Mon 2020-02-10 09:33:37 +03; 7s ago

Process: 23548 ExecStart=/usr/local/bin/vault_seal_manager.sh (code=exited, status=1/FAILURE)

Main PID: 23548 (code=exited, status=1/FAILURE)

Feb 10 09:33:37 ubuntu systemd[1]: Started Vault Initialization and Seal Manager.

Feb 10 09:33:37 ubuntu vault_seal_manager.sh[23548]: Error checking seal status: Get https://127.0.0.1:8200/v1/sys/seal-status: x509: certificate signed by unknown authority

Feb 10 09:33:37 ubuntu vault_seal_manager.sh[23548]: VaultSealManager-[09:33:37 AM 02/10/2020]-[INFO]: Got unseal keys successfull…

Feb 10 09:33:37 ubuntu vault_seal_manager.sh[23548]: VaultSealManager-[09:33:37 AM 02/10/2020]-[INFO]: Attempting to unseal local Vault instance with acquired unseal keys…

Feb 10 09:33:37 ubuntu vault_seal_manager.sh[23548]: VaultSealManager-[09:33:37 AM 02/10/2020]-[ERROR]: Error unsealing local Vault instance!

Feb 10 09:33:37 ubuntu systemd[1]: vault-seal-manager.service: Main process exited, code=exited, status=1/FAILURE

Feb 10 09:33:37 ubuntu systemd[1]: vault-seal-manager.service: Unit entered failed state.

Feb 10 09:33:37 ubuntu systemd[1]: vault-seal-manager.service: Failed with result ‘exit-code’.

At the same time, when we look at the vault.service stus in this case, there is an error as follows:

root@ubuntu:/home/robust2# systemctl status vault.service
● vault.service - vault server
Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-02-10 09:35:10 +03; 21s ago
Process: 23917 ExecStop=/usr/local/bin/vault operator step-down (code=exited, status=2)
Main PID: 24032 (vault)
Tasks: 15
Memory: 10.5M
CPU: 172ms
CGroup: /system.slice/vault.service
└─24032 /usr/local/bin/vault server -config=/etc/vault.d/vault.hcl

Feb 10 09:35:10 ubuntu vault[24032]: Storage: file
Feb 10 09:35:10 ubuntu vault[24032]: Version: Vault v1.3.0
Feb 10 09:35:10 ubuntu vault[24032]: ==> Vault server started! Log data will stream in below:
Feb 10 09:35:10 ubuntu vault[24032]: 2020-02-10T09:35:10.237+0300 [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
Feb 10 09:35:10 ubuntu vault[24032]: 2020-02-10T09:35:10.237+0300 [WARN] no api_addr value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually
Feb 10 09:35:22 ubuntu vault[24032]: 2020-02-10T09:35:22.325+0300 [INFO] http: TLS handshake error from 127.0.0.1:42230: remote error: tls: bad certificate
Feb 10 09:35:22 ubuntu vault[24032]: 2020-02-10T09:35:22.477+0300 [INFO] http: TLS handshake error from 127.0.0.1:42232: remote error: tls: bad certificate
Feb 10 09:35:22 ubuntu vault[24032]: 2020-02-10T09:35:22.635+0300 [INFO] http: TLS handshake error from 127.0.0.1:42234: remote error: tls: bad certificate
Feb 10 09:35:22 ubuntu vault[24032]: 2020-02-10T09:35:22.777+0300 [INFO] http: TLS handshake error from 127.0.0.1:42236: remote error: tls: bad certificate
Feb 10 09:35:22 ubuntu vault[24032]: 2020-02-10T09:35:22.899+0300 [INFO] http: TLS handshake error from 127.0.0.1:42238: remote error: tls: bad certificate

While in this state, I can’t find where my fault is. While I can run the script without any trouble from the command line, when I run it through systemd;
http: TLS handshake error form xxx remote error: tls: bad cetificate
What is the reason I got the error?

Hi there!

This indicates that the Vault CLI couldn’t validate the used certificate from your Vault instance because your certificate is not trusted by the host OS.
You can set VAULT_CACERT(https://www.vaultproject.io/docs/commands/#vault_cacert) in your script to the path of your self-signed certificate which should solve your problem.

Another option would be to add your CA certificate to your host OS certificate store which is OS dependent.

Cheers,
Michel

1 Like

Hi @michelvocks

I added the OS certs of my vault self-signed certificate file and my problem solved with your advice.
Thank you