Vault HA in kubernetes with postgres storage

Vault
1

I am working with an official docker image of vault version 1.12.0.
Deploying in kubernetes using helm.
I am using postgres as storage in side the kuberentes.

My configuration (vault.json) is below:
{
“listener”: [{
“tcp”: {
“address” : “127.0.0.1:8120”,
“tls_disable” : 1
}
}],
“api_addr”: “http://127.0.0.1:8120”,
“storage”: {
“postgresql”: {
“connection_url”: “postgres://vault:vault@postgresXXXX/xxxxx?sslmode=disable”,
“ha_enabled”: true,
“ha_table”: “vault_ha_locks”
}
},
“max_lease_ttl”: “10h”,
“default_lease_ttl”: “10h”,
“ui”:true,
“disable_mlock”:true
}

All the API’s are working as expected if there is only one instance.

I need multiple replica of vault.

I got one active node and standby nodes.

vault status - Active node

bash-5.1$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.12.0
Build Date      2022-10-10T18:14:33Z
Storage Type    postgresql
Cluster Name    vault-cluster-c3225009
Cluster ID      6c326406-f59e-e307-59fb-1eaf56a2ed9a
HA Enabled      true
HA Cluster      https://127.0.0.1:8121
HA Mode         active
Active Since    2022-11-15T11:10:07.242796984Z

vault status - Stand by node

bash-5.1$ vault status
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.12.0
Build Date             2022-10-10T18:14:33Z
Storage Type           postgresql
Cluster Name           vault-cluster-c3225009
Cluster ID             6c326406-f59e-e307-59fb-1eaf56a2ed9a
HA Enabled             true
HA Cluster             n/a
HA Mode                standby
Active Node Address    <none>

Here I am getting below error while hitting the API in standby node:

{"code":500,"message":"Error making API request.\n\nURL: GET http://127.0.0.1:8120/v1/transit/keys?list=true\nCode: 500. Errors:\n\n* local node not active but active cluster node not found"}

How to set the Active Node Address in standby node from kubernetes ?

Any help would be greatly appreciated.

2

The problem here is that you have specified the loopback IP address 127.0.0.1 as the address each Vault node advertises, for the other node to talk to it.

As a result, the standby node is unable to communicate with the active node.

3

Thank you so much for the reply.
Could you please help me to set the correct address in each nodes?
I have multiple Vault nodes in side the container.

INBLRCYBPKG4525:~ # kubectl get pods -A
NAMESPACE        NAME                                              READY   STATUS      RESTARTS          AGE
cp               secret-svc-7dc54f488b-5mkvg                       1/1     Running     0                 14m
cp               secret-svc-7dc54f488b-9mdmx                       1/1     Running     0                 14m
cp               secret-svc-7dc54f488b-h6c92                       1/1     Running     0                 14m
4

This is just personal preference, but I’d really try to avoid using JSON to configure Vault - yes, it works, but it’s horribly unreadable compared to HCL.

It seems you’re using Kubernetes and Helm, but via your own custom Helm charts, so I’m not sure what correct would look like in your environment.

However, I can tell you that the official Vault Helm chart makes use of Kubernetes field references to get the pod IP address into a variable:

and then uses that variable to set the VAULT_API_ADDR environment variable (which overrides what is written in the configuration file):

It does some other things to set VAULT_CLUSTER_ADDR too - though those things it does are specific to running Vault as a K8s StatefulSet, because that chart is aimed at running with Raft (Integrated Storage).

Hopefully you can pick and choose relevant bits of this to craft your own configuration.

1 Like
5

Thank you!
It helps a lot