Vault Failed to Start on New Installation of Integrated Storage with error code 203

Vault
1

Team, I am setting up a new vault environment using the integrated storage solution v1.4.
https://learn.hashicorp.com/vault/operations/ops-deployment-guide-raft.

When I try to start the Vault, it’s failing with error code #203.

$sudo systemctl status vault
● vault.service - “HashiCorp Vault - A tool for managing secrets”
Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2020-07-19 18:45:49 UTC; 175ms ago
Docs: https://www.vaultproject.io/docs/
Process: 39819 ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl (code=exited, status=203/EXEC)
Main PID: 39819 (code=exited, status=203/EXEC)

This is my config file /etc/vault.d/vault.hcl
listener “tcp” {
address = “0.0.0.0:8200”
tls_cert_file = “/home/user/vaultcrts/fullchain.pem”
tls_key_file = “/home/user/vaultcrts/privkey.pem”
}

seal “pkcs11” {
lib = “/usr/vault/lib/libCryptoki2_64.so”
slot = “0”
pin = “AAAA-BBBB-CCCC-DDDD”
key_label = “vault-hsm-key”
hmac_key_label = “vault-hsm-hmac-key”
}

storage “raft” {
path = “/opt/raft”
node_id = “raft_node_1”
}

api_addr = “http://127.0.0.1:8200”
ui = true.

Need Help On Fixing this as we need to give a demo

2

Any log/ debug output?

The listener address looks weird. I don’t think you can bind to 0.0.0.0 for multiple network interfaces with only one entry.

The listener stanza may be specified more than once to make Vault listen on multiple interfaces.

Why should Vault listening on all interfaces? I don’t think this would be that secure.

3

@Wolfsrudel Thanks for the prompt response.
As per the suggested I’ve changed it to use wit/etc/vault.d/vault.hclh a single interface loop back in the config file

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_cert_file = "/home/MP7652/vaultcrts/fullchain.pem"
  tls_key_file  = "/home/MP7652/vaultcrts/privkey.pem"
}

I’m still not able to start the service and it’s failing with exit code 203

$ sudo systemctl status vault
● vault.service - “HashiCorp Vault - A tool for managing secrets”
Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2020-07-20 06:30:50 UTC; 1min 28s ago
Docs: https://www.vaultproject.io/docs/
Process: 1615 ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl (code=exited, status=203/EXEC)
Main PID: 1615 (code=exited, status=203/EXEC)

Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Service RestartSec=5s expired, scheduling restart.
Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 3.
Jul 20 06:30:50 vault-2 systemd[1]: Stopped “HashiCorp Vault - A tool for managing secrets”.
Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Start request repeated too quickly.
Jul 20 06:30:50 vault-2 systemd[1]: vault.service: Failed with result ‘exit-code’.
Jul 20 06:30:50 vault-2 systemd[1]: Failed to start “HashiCorp Vault - A tool for managing secrets”.

And the below is my complete vault config file:
listener “tcp” {
address = “127.0.0.1:8200”
tls_cert_file = “/home/MP7652/vaultcrts/fullchain.pem”
tls_key_file = “/home/MP7652/vaultcrts/privkey.pem”
}

seal “pkcs11” {
lib = “/usr/vault/lib/libCryptoki2_64.so”
slot = “0”
pin = “AAAA-BBBB-CCCC-DDDD”
key_label = “vault-hsm-key”
hmac_key_label = “vault-hsm-hmac-key”
}

storage “raft” {
path = “/opt/raft”
node_id = “raft_node_1”
}

api_addr = “http://127.0.0.1:8200”
ui = true.

Journalctl log

Jul 20 06:28:11 vault-2 systemd[1]: Failed to start "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:35 vault-2 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:35 vault-2 systemd[1]: vault.service: Main process exited, code=exited, status=203/EXEC
Jul 20 06:30:35 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Service RestartSec=5s expired, scheduling restart.
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 1.
Jul 20 06:30:40 vault-2 systemd[1]: Stopped "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:40 vault-2 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Main process exited, code=exited, status=203/EXEC
Jul 20 06:30:40 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Service RestartSec=5s expired, scheduling restart.
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 2.
Jul 20 06:30:45 vault-2 systemd[1]: Stopped "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:45 vault-2 systemd[1]: Started "HashiCorp Vault - A tool for managing secrets".
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Main process exited, code=exited, status=203/EXEC
Jul 20 06:30:45 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.

Please let me know if something needs to be tuned.
Thanks

4

Hello,

Sorry for the trouble you are having with Vault !

Is the Vault binary you are trying to run a Vault Enterprise one ? The Vault OSS binary does not support the seal pkcs11 stanza, would you verify this for me please.

Also, can you verify the seal stanza is correct, it needs to be, cause Vault is going to try to decrypt its master key with this HSM. If you are not using HSM, you can go ahead and remove the seal stanza, this way Vault is going to use regular SSS.

Martin

1 Like
5

Hi Martin

I am new to vault and I don’t think I am using Vault Enterprise.
As per your suggestion, I will remove the seal stanza and gave a try and it’s still failing.

listener “tcp” {
address = “127.0.0.1:8200”
tls_cert_file = “/home/user/vaultcrts/fullchain.pem”
tls_key_file = “/home/user/vaultcrts/privkey.pem”
}
storage “raft” {
path = “/opt/raft”
node_id = “raft_node_1”
}
api_addr = “http://127.0.0.1:8200”
ui = true

Jul 20 07:00:23 vault-2 sudo[1784]:   MP7652 : TTY=pts/0 ; PWD=/home/MP7652 ; USER=root ; COMMAND=/bin/systemctl status vault
Jul 20 07:00:23 vault-2 sudo[1784]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jul 20 07:00:23 vault-2 sudo[1784]: pam_unix(sudo:session): session opened for user root by MP7652(uid=0)
Jul 20 07:00:23 vault-2 sudo[1784]: pam_unix(sudo:session): session closed for user root
Jul 20 07:00:42 vault-2 sudo[1788]:   MP7652 : TTY=pts/0 ; PWD=/home/MP7652 ; USER=root ; COMMAND=/bin/systemctl stop vault
Jul 20 07:00:42 vault-2 sudo[1788]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jul 20 07:00:42 vault-2 sudo[1788]: pam_unix(sudo:session): session opened for user root by MP7652(uid=0)
Jul 20 07:00:42 vault-2 sudo[1788]: pam_unix(sudo:session): session closed for user root
Jul 20 07:00:45 vault-2 sudo[1792]:   MP7652 : TTY=pts/0 ; PWD=/home/MP7652 ; USER=root ; COMMAND=/bin/systemctl start vault
Jul 20 07:00:45 vault-2 sudo[1792]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jul 20 07:00:45 vault-2 sudo[1792]: pam_unix(sudo:session): session opened for user root by MP7652(uid=0)
Jul 20 07:00:45 vault-2 systemd[1]: vault.service: Start request repeated too quickly.
Jul 20 07:00:45 vault-2 systemd[1]: vault.service: Failed with result 'exit-code'.
Jul 20 07:00:45 vault-2 systemd[1]: Failed to start "HashiCorp Vault - A tool for managing secrets".
-- Subject: Unit vault.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Unit vault.service has failed.
-- 
-- The result is failed.

Let me know if something needs to be trimmed on config.

Thanks

6

Hello,

For getting started with Vault i would suggest running it in dev mode, here is how i do it :

vault server -dev -dev-listen-address=127.0.0.1:8200 --log-level=TRACE -dev-root-token-id=root

In order for this command to run you need to have the vault binary inside your PATH variable, it will start a Vault server with inmem (all data is stored in memory and lost after restart) storage and root token set to value root.

If you still like to run Vault with Raft storage, i would suggest to start with simple config without TLS, for example :

 # Storage config
storage "raft" {
  path = "/path/to/raft/data" # Set this to directory with R/W access
  node_id = "raft_node_1" # Pick a name for the node
}
cluster_addr = "http://127.0.0.1:8201" # This address is used for node to node communication 
api_addr = "http://127.0.0.1:8200" # Address of the API for Vault

# Listener config
listener "tcp" {
address = "0.0.0.0:8200" # Listen on all available interfaces
tls_disable = 1 # No TLS
}

Martin

7

Previously I have successfully integrated vault with HSM which was password enabled and was up and running.
I am at remote location connected to network , trying to start Vault Server integrated with Ped enabled HSM device.
I have added HSM configurations in vault Config file but didn’t know what should I set pin inorder to access HSM from Vault server.
I have tried entering admin pwd, Crypto officer pin and ped device attached to the HSM but still I am unable to start vault.

error parsing Seal configuration: error fetching session to test HSM key configuration: error logging in to HSM: pkcs11: 0x80000028
2021-07-13T05:39:54.367-0400 [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""

config:
{
“pkcs11”: {
“generate_key”: “true”,
“hmac_key_label”: “HashiCorp_hmac”,
“key_label”: “HashiCorp”,
“lib”: “/opt/vault/hsm/libCryptoki2_64.so”,
“pin”: “xxxxx”,
“slot”: “1”
}
}
]
Any Suggestions please!..