Vault Failed to Start on New Installation of Integrated Storage with error code 203

Team, I am setting up a new vault environment using the integrated storage solution v1.4.
https://learn.hashicorp.com/vault/operations/ops-deployment-guide-raft.

When I try to start the Vault, it’s failing with error code #203.

$sudo systemctl status vault
● vault.service - “HashiCorp Vault - A tool for managing secrets”
Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2020-07-19 18:45:49 UTC; 175ms ago
Docs: https://www.vaultproject.io/docs/
Process: 39819 ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl (code=exited, status=203/EXEC)
Main PID: 39819 (code=exited, status=203/EXEC)

This is my config file /etc/vault.d/vault.hcl
listener “tcp” {
address = “0.0.0.0:8200”
tls_cert_file = “/home/user/vaultcrts/fullchain.pem”
tls_key_file = “/home/user/vaultcrts/privkey.pem”
}

seal “pkcs11” {
lib = “/usr/vault/lib/libCryptoki2_64.so”
slot = “0”
pin = “AAAA-BBBB-CCCC-DDDD”
key_label = “vault-hsm-key”
hmac_key_label = “vault-hsm-hmac-key”
}

storage “raft” {
path = “/opt/raft”
node_id = “raft_node_1”
}

api_addr = “http://127.0.0.1:8200
ui = true.

Need Help On Fixing this as we need to give a demo

Any log/ debug output?

The listener address looks weird. I don’t think you can bind to 0.0.0.0 for multiple network interfaces with only one entry.

The listener stanza may be specified more than once to make Vault listen on multiple interfaces.

Why should Vault listening on all interfaces? I don’t think this would be that secure.

@Wolfsrudel Thanks for the prompt response.
As per the suggested I’ve changed it to use wit/etc/vault.d/vault.hclh a single interface loop back in the config file

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_cert_file = "/home/MP7652/vaultcrts/fullchain.pem"
  tls_key_file  = "/home/MP7652/vaultcrts/privkey.pem"
}

I’m still not able to start the service and it’s failing with exit code 203

$ sudo systemctl status vault
● vault.service - “HashiCorp Vault - A tool for managing secrets”
Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2020-07-20 06:30:50 UTC; 1min 28s ago
Docs: https://www.vaultproject.io/docs/
Process: 1615 ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl (code=exited, status=203/EXEC)
Main PID: 1615 (code=exited, status=203/EXEC)

Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Service RestartSec=5s expired, scheduling restart.
Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 3.
Jul 20 06:30:50 vault-2 systemd[1]: Stopped “HashiCorp Vault - A tool for managing secrets”.
Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Start request repeated too quickly.
Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Failed with result ‘exit-code’.
Jul 20 06:30:50 vault-2 systemd[1]: Failed to start “HashiCorp Vault - A tool for managing secrets”.

And the below is my complete vault config file:
listener “tcp” {
address = “127.0.0.1:8200”
tls_cert_file = “/home/MP7652/vaultcrts/fullchain.pem”
tls_key_file = “/home/MP7652/vaultcrts/privkey.pem”
}

seal “pkcs11” {
lib = “/usr/vault/lib/libCryptoki2_64.so”
slot = “0”
pin = “AAAA-BBBB-CCCC-DDDD”
key_label = “vault-hsm-key”
hmac_key_label = “vault-hsm-hmac-key”
}

storage “raft” {
path = “/opt/raft”
node_id = “raft_node_1”
}

api_addr = “http://127.0.0.1:8200
ui = true.

Journalctl log

Jul 20 06:28:11 vault-2 systemd[1]: Failed to start "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:35 vault-2 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:35 vault-2 systemd[1]: vault.service: Main process exited, code=exited, status=203/EXEC
Jul 20 06:30:35 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Service RestartSec=5s expired, scheduling restart.
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 1.
Jul 20 06:30:40 vault-2 systemd[1]: Stopped "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:40 vault-2 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Main process exited, code=exited, status=203/EXEC
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Service RestartSec=5s expired, scheduling restart.
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 2.
Jul 20 06:30:45 vault-2 systemd[1]: Stopped "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:45 vault-2 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Main process exited, code=exited, status=203/EXEC
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.


Please let me know if something needs to be tuned.
Thanks

Hello,

Sorry for the trouble you are having with Vault !

Is the Vault binary you are trying to run a Vault Enterprise one ? The Vault OSS binary does not support the seal pkcs11 stanza, would you verify this for me please.

Also, can you verify the seal stanza is correct, it needs to be, cause Vault is going to try to decrypt its master key with this HSM. If you are not using HSM, you can go ahead and remove the seal stanza, this way Vault is going to use regular SSS.

Martin

1 Like

Hi Martin

I am new to vault and I don’t think I am using Vault Enterprise.
As per your suggestion, I will remove the seal stanza and gave a try and it’s still failing.

listener “tcp” {
address = “127.0.0.1:8200”
tls_cert_file = “/home/user/vaultcrts/fullchain.pem”
tls_key_file = “/home/user/vaultcrts/privkey.pem”
}
storage “raft” {
path = “/opt/raft”
node_id = “raft_node_1”
}
api_addr = “http://127.0.0.1:8200
ui = true

Jul 20 07:00:23 vault-2 sudo[1784]:   MP7652 : TTY=pts/0 ; PWD=/home/MP7652 ; USER=root ; COMMAND=/bin/systemctl status vault
Jul 20 07:00:23 vault-2 sudo[1784]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jul 20 07:00:23 vault-2 sudo[1784]: pam_unix(sudo:session): session opened for user root by MP7652(uid=0)
Jul 20 07:00:23 vault-2 sudo[1784]: pam_unix(sudo:session): session closed for user root
Jul 20 07:00:42 vault-2 sudo[1788]:   MP7652 : TTY=pts/0 ; PWD=/home/MP7652 ; USER=root ; COMMAND=/bin/systemctl stop vault
Jul 20 07:00:42 vault-2 sudo[1788]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jul 20 07:00:42 vault-2 sudo[1788]: pam_unix(sudo:session): session opened for user root by MP7652(uid=0)
Jul 20 07:00:42 vault-2 sudo[1788]: pam_unix(sudo:session): session closed for user root
Jul 20 07:00:45 vault-2 sudo[1792]:   MP7652 : TTY=pts/0 ; PWD=/home/MP7652 ; USER=root ; COMMAND=/bin/systemctl start vault
Jul 20 07:00:45 vault-2 sudo[1792]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jul 20 07:00:45 vault-2 sudo[1792]: pam_unix(sudo:session): session opened for user root by MP7652(uid=0)
Jul 20 07:00:45 vault-2 systemd[1]: vault.service: Start request repeated too quickly.
Jul 20 07:00:45 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.
Jul 20 07:00:45 vault-2 systemd[1]: Failed to start "HashiCorp Vault - A tool for managing secrets".
-- Subject: Unit vault.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Unit vault.service has failed.
-- 
-- The result is failed.

Let me know if something needs to be trimmed on config.

Thanks

Hello,

For getting started with Vault i would suggest running it in dev mode, here is how i do it :

vault server -dev -dev-listen-address=127.0.0.1:8200 --log-level=TRACE -dev-root-token-id=root

In order for this command to run you need to have the vault binary inside your PATH variable, it will start a Vault server with inmem (all data is stored in memory and lost after restart) storage and root token set to value root.

If you still like to run Vault with Raft storage, i would suggest to start with simple config without TLS, for example :

 # Storage config
storage "raft" {
  path = "/path/to/raft/data" # Set this to directory with R/W access
  node_id = "raft_node_1" # Pick a name for the node
}
cluster_addr = "http://127.0.0.1:8201" # This address is used for node to node communication 
api_addr = "http://127.0.0.1:8200" # Address of the API for Vault

# Listener config
listener "tcp" {
address = "0.0.0.0:8200" # Listen on all available interfaces
tls_disable = 1 # No TLS
}

Martin