Hey,
I am trying to install my custom plugin to a Vault instance and through the Vault’s CLI. I’m also using Helm.
Config:
config: |
ui = true
listener "tcp" {
tls_disable = "false"
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
tls_key_file = "/vault/userconfig/vault-tls/tls.key"
tls_require_and_verify_client_cert = "false"
tls_disable_client_certs = "false"
}
storage "file" {
path = "/vault/data"
}
While the plugin works without TLS, when the latter is enabled, I’m getting the errors you can see below:
2021-11-17T19:28:50.694Z [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h4e42e7418b0785dfd20130f284ea8df13c967097875734a253f7e15dd5d9d782
2021-11-17T19:28:50.695Z [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h8c9f874c91159299b4bda67a8eabd5f9ade047083300eaf308080d621db3f70e
2021-11-17T19:28:50.696Z [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h5b2bb9d1538bdecc3093c8a9b0fe307024e063c525c18602ec08caf6ad5d8286
2021-11-17T19:28:50.697Z [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h7423af481494a75f50b4d21e6307a55783732d18ca2d17a930b0781fc54c6c87
2021-11-17T19:28:54.662Z [INFO] http: TLS handshake error from 10.1.20.72:39060: remote error: tls: bad certificate
2021-11-17T19:28:55.948Z [INFO] http: TLS handshake error from 10.1.20.72:39074: remote error: tls: bad certificate
2021-11-17T19:28:58.468Z [INFO] http: TLS handshake error from 10.1.20.72:39098: remote error: tls: bad certificate
2021-11-17T19:28:58.470Z [ERROR] secrets.customPlugin.customPlugin_3c93b6e4.customPlugin.customPlugin: plugin tls init: error=“error during token unwrap request: Put “https://10.1.20.72:8200/v1/sys/wrapping/unwrap”: x509: cannot validate certificate for 10.1.20.72 because it doesn’t contain any IP SANs” timestamp=2021-11-17T19:28:58.468Z
2021-11-17T19:28:58.470Z [ERROR] rollback: error rolling back: path=customPlugin_engine/
error=
| Unrecognized remote plugin message:
|
| This usually means that the plugin is either invalid or simply
| needs to be recompiled to support the latest protocol.
2021-11-17T19:28:58.470Z [ERROR] secrets.system.system_fd695f65: unmount failed: path=some_path/
error=
| Unrecognized remote plugin message:
|
| This usually means that the plugin is either invalid or simply
| needs to be recompiled to support the latest protocol.
While I understand what Vault tells me, I can’t find any solution to the issue.
I have tried everything I could think of/find, however, I always get the same error, or errors similar to it.
This is how I’m starting the instance:
CA_BUNDLE=$(kubectl get secret vault-tls -n vault -o “jsonpath={ .data[‘ca.crt’] }” | base64)
helm install vault custom-vault
-n vault
–set=“injector.certs.secretName=vault-tls”
–set=“injector.certs.caBundle=${CA_BUNDLE?}”
Any idea will be much appreciated!
Thanks!
Update
It seems that the Vault is not using the provided TLS Certificate.
When I:
openssl s_client -connect vault_server_ip:8200
I get:
subject=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
issuer=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
However, everything seems fine. The Vault server seems to load the correct files:
“tls_cert_file”:“/vault/userconfig/vault-tls/tls.crt”,“tls_client_ca_file”:“/vault/userconfig/vault-tls/ca.crt”,“tls_disable”:“false”,“tls_key_file”:“/vault/userconfig/vault-tls/tls.key”}
The server:extraEnvironmentVars:
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt
VAULT_TLSCERT: /vault/userconfig/vault-tls/tls.crt
VAULT_TLSKEY: /vault/userconfig/vault-tls/tls.key
As the subject says the certificate you are seeing is from an ingress controller. So it sounds like you have configured it to use an ingress that terminates the traffic itself, rather than something which passes traffic directly to the pod.
Hello stuart,
thank you for your support.
Managed to fix the ingress issue. Something with the configuration was indeed wrong.
However, I still can’t install the custom plugin.
==> Vault server configuration:
Api Address: https://10.1.20.98:8200
Cgo: disabled
Cluster Address: https://vault-0.vault-internal:8201
Go Version: go1.16.2
Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: debug
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: file
Version: Vault v1.8.0-dev
2021-11-18T14:35:57.727Z [DEBUG] secrets.customPlugin.customPlugin_461f4388.customPlugin: starting plugin: path=/vault/plugins/customPlugin args=[“/vault/plugins/customPlugin”]
2021-11-18T14:35:57.728Z [DEBUG] secrets.customPlugin.customPlugin_461f4388.customPlugin: plugin started: path=/vault/plugins/customPlugin pid=146
2021-11-18T14:35:57.728Z [DEBUG] secrets.customPlugin.customPlugin_461f4388.customPlugin: waiting for RPC address: path=/vault/plugins/customPlugin
2021-11-18T14:35:57.736Z [INFO] http: TLS handshake error from 10.1.20.98:59954: remote error: tls: bad certificate
2021-11-18T14:35:58.959Z [INFO] http: TLS handshake error from 10.1.20.98:59978: remote error: tls: bad certificate
2021-11-18T14:36:01.054Z [INFO] http: TLS handshake error from 10.1.20.98:60000: remote error: tls: bad certificate
2021-11-18T14:36:01.055Z [ERROR] secrets.customPlugin.customPlugin_461f4388.customPlugin.customPlugin: plugin tls init: error=“error during token unwrap request: Put “https://10.1.20.98:8200/v1/sys/wrapping/unwrap”: x509: cannot validate certificate for 10.1.20.98 because it doesn’t contain any IP SANs” timestamp=2021-11-18T14:36:01.054Z
2021-11-18T14:36:01.056Z [DEBUG] secrets.customPlugin.customPlugin_461f4388.customPlugin: plugin process exited: path=/vault/plugins/customPlugin pid=146
2021-11-18T14:36:01.056Z [ERROR] rollback: error rolling back: path=secret_customPlugin_engine/
error=
| Unrecognized remote plugin message:
|
| This usually means that the plugin is either invalid or simply
| needs to be recompiled to support the latest protocol.
2021-11-18T14:36:01.056Z [ERROR] secrets.system.system_d0b5cd86: unmount failed: path=secret_customPlugin_engine/
error=
| Unrecognized remote plugin message:
|
| This usually means that the plugin is either invalid or simply
| needs to be recompiled to support the latest protocol.
The issue seems to be that it’s trying to reach Vault through the Vault’s IP address (which is not included in the Certificate, as the error suggests). I have tried to export the VAULT_ADDR to, e.g.,
export VAULT_ADDR=“https://vault:8200”
but it’s keep using the Vault’s IP address.
Update #2
As I earlier said, the problem stems from the fact that Vault uses its Api Address (IP) for every action. I noticed that it was reusing the same Api Addresses on every initialization (e.g. 10.1.20.xx), so I included these addresses to the PEM Certificate, and, yes, it worked. I was able to install the plugin.
So, how can I solve this issue?
These addresses are not standard, thus I cannot include them to the certificate.
Can I somehow use a domain as an Api Address?
Edit:
Appending the FQDN (that is included in the X509 Certificate) to the injector.extraEnvironmentVars, did the trick:
extraEnvironmentVars:
VAULT_API_ADDR: “https://FQDN”