Vault Fails to start

Vault
1

I have setup Hashicorp - vault (Vault v1.5.4) on Ubuntu 18.04. My backend is Consul (single node running on same server as vault) - consul service is up. (Also tried local file storage instead of Consul). I have the basic/default configuration.

/etc/vault/config.json

{
“listener”: [{
“tcp”: {
“address” : “0.0.0.0:8200”,
“tls_disable” : 1
}
}],

“api_addr”: “http://:8200”,

“storage”: “file”: {
“path” : “/opt/vault/data”
},

“max_lease_ttl”: “10h”,
“default_lease_ttl”: “10h”,
“ui”:true,
“log_level”: “Debug”
}

/etc/systemd/system/vault.service

[Unit]
Description=vault service
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault/config.json

[Service]
User=vault
Group=vault
EnvironmentFile=-/etc/sysconfig/vault
Environment=GOMAXPROCS=2
Restart=on-failure
ExecStart=/opt/vault/vault server -config=/etc/vault/config.json
StandardOutput=/opt/vault/logs/output.log
StandardError=/opt/vault/logs/error.log
LimitMEMLOCK=infinity
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

My vault service fails to start

systemctl status vault
● vault.service - vault service
Loaded: loaded (/etc/systemd/system/vault.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-10-06 11:06:59 CDT; 34s ago
Process: 13236 ExecStart=/opt/vault/vault server -config=/etc/vault/config.json (code=exited, status=203/EXEC)

systemctl list-units --type=service | grep "vault"
vault.service                        loaded failed failed  vault service  

journalctl -xe -u vault
ubuntu2 systemd[1]: vault.service: Scheduled restart job, restart counter is at 5.
 ubuntu2 systemd[1]: vault.service: Start request repeated too quickly.
ubuntu2 systemd[1]: vault.service: Failed with result 'exit-code'.
ubuntu2 systemd[1]: Failed to start vault service.

There is nothing in the log files output or error.

How do I debug this?

2

Only way to know what its trying to do is if you share your config file…

3

Updated with config details. The Api address in post ‘appears’ as “api_addr”: “http://:8200” - but does have the correct IP in the config

4

What if you just run the server from the console? ie
$ vault server -config=/etc/vault/config.json
Same thing?

5

vault server -config=/etc/vault/config.json
error loading configuration from /etc/vault/config.json: error parsing ‘storage’: storage.storage: At -: root: not an object type for map (*ast.LiteralType)

My storage in config.json is and I do have this directory /opt/vault/data:
“storage”: “file”: {
“path” : “/opt/vault/data”
},

6

Yeah that’s just invalid syntax.
I don’t use json, but I bet it should be something like

"storage": {
    "file": {
    "path" : "/foo"
    }
 },

but the docs for sure have an HCL example

7

Ok, fixed that. That issue probably has crept hcl and json - sometimes the examples get mingled (!!). (also multiple cut-pastes consul/file …)

Next issue now mlock.
Error initializing core: Failed to lock memory: cannot allocate memory.
So the options are: disable mlock on vault or setup mlcok on the machine?
Should I just add "disable_mlock": true

8

Ok disbaled mlcok - it started up. But the ui does not open http://:8200.
Let me check the ip tables. After I fixed ip tables the vault ui opens up.

Thanks much.
(Have to fix consul - its ui doesn’t not show up- could be same iptables issue)

9

But I still cannot start vault as service . Same error
sudo systemctl reload-or-restart vault
sudo systemctl status vault
Active: failed (Result: exit-code) since Tue 2020-10-06 13:11:16 CDT; 26s ago
Process: 13507 ExecStart=/opt/vault/vault server -config=/etc/vault/config.json (code=exited, status=203/EXEC)
Main PID: 13507 (code=exited, status=203/EXEC)
vault.service: Main process exited, code=exited, status=203/EXEC
vault.service: Failed with result ‘exit-code’.
vault.service: Service hold-off time over, scheduling restart.
vault.service: Scheduled restart job, restart counter is at 5.
Stopped vault service.
vault.service: Start request repeated too quickly.
vault.service: Failed with result ‘exit-code’.
Failed to start vault service.

10

Can you check if the executable file vault has the right permissions?

/Alf

11

Have you thought about using the yum/rpm packages to install?
Check your permissions, systemd config, etc

12

My permissions appear OK(?)

ls -l /opt/vault
drwxrwxr-x 2 vault vault 4096 Oct 2 14:12 bin
drwxrwxr-x 2 vault vault 4096 Oct 2 13:38 data
drwxrwxr-x 2 vault vault 4096 Oct 2 13:38 logs

/opt/vault/bin$ ls -l
-rwxrwxr-x 1 vault vault 138547711 Sep 23 16:08 vault

ls -l /etc/vault
-rw-r--r-- 1 vault vault 341 Oct 6 12:57 config.json

[Service]
User=vault
Group=vault

less /etc/passwd | grep "vault"
vault:x:999:998::/etc/vault:/bin/false

13

I fixed permissions on config.json +x - not sure if that was needed, but it still fails
ls -l /etc/vault
-rwxr-xr-- 1 vault vault 341 Oct 6 12:57 config.json

14

Yeah, I had followed something like this to setup vault, my permissions seem fine - posted - can you validate.

15

I gutted my install and used the exact same config Vault Deployment tutorial | Vault - HashiCorp Learn. This worked.

Who knows what the problem with the original was - permissions/whatever - no idea.

I could init/unseal and do some basic stuff.

But sealing failed
$ vault operator seal
Error sealing: Error making API request.

URL: PUT http://192.168.37.128:8200/v1/sys/seal
Code: 500. Errors:

* 1 error occurred:
        * missing client token

What is this client token, the docs have no mention.

16

Are you auth’d with a token, ie
vault login ?

17

I simply initialized and unsealed the vault.

I did export VAULT_TOKEN="<your-token>" and vault auth enable approle - some stuff.

After that I vault operator seal. What else is required to be done?

18

Ok, I logged in with the “token” and then ran the unseal it worked. I will setup users etc and figure all that out.

Cant find a logout method(?)

Thanks

19

Why do you want to seal Vault at runtime? Shut it down, it’s sealed.

For logging out: exit your session and remove the stored token. There won’t be any logout:

This seems to be a hard one…

20

Whenever there is a login - there should be a logout. Else, how do I end my session? Do I kill my putty session- that likely may not kill my vault session. Deleting token etc can be too error prone/risky.

I don’t see a vault stop CLI command. Are you referring to stopping the service? If a vault is unsealed - could it cause corruption?

Assuming when I do a vault auth/login - it is just for my user and session not any other user.

Yes unsealed vault serves multiple users - so leaving it unsealed would be necessary.