maxb
August 18, 2023, 7:41pm
2
This issue came up in the Vault issue tracker before:
opened 08:29AM - 21 Apr 23 UTC
closed 07:31PM - 14 Jul 23 UTC
ecosystem/plugin
k8s
<!-- Please reserve GitHub issues for bug reports and feature requests.
For q… uestions, the best place to get answers is on our [discussion forum](https://discuss.hashicorp.com/c/vault), as they will get more visibility from experienced users than the issue tracker.
Please note: We take Vault's security and our users' trust very seriously. If you believe you have found a security issue in Vault, please responsibly disclose by contacting us at security@hashicorp.com. Our PGP key is available at [our security page](https://www.hashicorp.com/security/).
-->
**Describe the bug**
I try to register a custom plugin i wrote. I've built a OCI image based on `vault:1.12.5` and added my custom plugin.
```console
$ kubectl exec -it -n vault vault-0 -c vault -- sha256sum /etc/vault/vault_plugins/vault-plugin-secrets-nats
13c753a26991858faf820604c6422c31e49368481a18335c6540ac28a7ce2aac /etc/vault/vault_plugins/vault-plugin-secrets-nats
$ vault plugin info secret vault-plugin-secrets-nats
Key Value
--- -----
args [--tls-skip-verify --ca-cert=/vault/tls/ca.crt]
builtin false
command vault-plugin-secrets-nats
deprecation_status n/a
name vault-plugin-secrets-nats
sha256 13c753a26991858faf820604c6422c31e49368481a18335c6540ac28a7ce2aac
version n/a
```
However, when i log at the logs of my vault instance i get errors like:
```
2023-04-21T08:17:38.476Z [INFO] core: successfully setup plugin catalog: plugin-directory=/etc/vault/vault_plugins
2023-04-21T08:17:38.479Z [INFO] core: successfully mounted: type=system version="v1.12.5+builtin.vault" path=sys/ namespace="ID: root. Path: "
2023-04-21T08:17:38.479Z [INFO] core: successfully mounted: type=identity version="v1.12.5+builtin.vault" path=identity/ namespace="ID: root. Path: "
2023-04-21T08:17:38.480Z [DEBUG] core: spawning a new plugin process: plugin_name=vault-plugin-secrets-nats id=Ed8aZMet4m
2023-04-21T08:17:38.690Z [ERROR] core: failed to create mount entry: path=nats-secrets/
error=
| invalid backend version: 2 errors occurred:
| \t* checksums did not match
| \t* checksums did not match
|
2023-04-21T08:17:38.691Z [WARN] core: skipping plugin-based mount entry: path=nats-secrets/
2023-04-21T08:17:38.691Z [INFO] core: successfully mounted: type=vault-plugin-secrets-nats version="" path=nats-secrets/ namespace="ID: root. Path: "
```
After deleting the pod three times this worked.
```
2023-04-21T08:30:48.504Z [INFO] core: upgrading plugin information: plugins=[]
2023-04-21T08:30:48.504Z [INFO] core: successfully setup plugin catalog: plugin-directory=/etc/vault/vault_plugins
2023-04-21T08:30:48.506Z [INFO] core: successfully mounted: type=system version="v1.12.5+builtin.vault" path=sys/ namespace="ID: root. Path: "
2023-04-21T08:30:48.508Z [INFO] core: successfully mounted: type=identity version="v1.12.5+builtin.vault" path=identity/ namespace="ID: root. Path: "
2023-04-21T08:30:48.509Z [DEBUG] core: spawning a new plugin process: plugin_name=vault-plugin-secrets-nats id=UPBhZnUJ7I
2023-04-21T08:30:48.588Z [INFO] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: configuring client automatic mTLS
2023-04-21T08:30:48.660Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: starting plugin: path=/etc/vault/vault_plugins/vault-plugin-secrets-nats args=["/etc/vault/vault_plugins/vault-plugin-secrets-nats", "--tls-skip-verify", "--ca-cert=/vault/tls/ca.crt"]
2023-04-21T08:30:48.660Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: plugin started: path=/etc/vault/vault_plugins/vault-plugin-secrets-nats pid=29
2023-04-21T08:30:48.660Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: waiting for RPC address: path=/etc/vault/vault_plugins/vault-plugin-secrets-nats
2023-04-21T08:30:48.671Z [ERROR] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats.vault-plugin-secrets-nats: plugin tls init: error="error parsing wrapping token: square/go-jose: compact JWS format must have three parts" timestamp=2023-04-21T08:30:48.670Z
2023-04-21T08:30:48.674Z [INFO] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: plugin process exited: path=/etc/vault/vault_plugins/vault-plugin-secrets-nats pid=29
2023-04-21T08:30:48.787Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: starting plugin: metadata=true path=/etc/vault/vault_plugins/vault-plugin-secrets-nats args=["/etc/vault/vault_plugins/vault-plugin-secrets-nats", "--tls-skip-verify", "--ca-cert=/vault/tls/ca.crt"]
2023-04-21T08:30:48.788Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: plugin started: metadata=true path=/etc/vault/vault_plugins/vault-plugin-secrets-nats pid=33
2023-04-21T08:30:48.788Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: waiting for RPC address: metadata=true path=/etc/vault/vault_plugins/vault-plugin-secrets-nats
2023-04-21T08:30:48.796Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats.vault-plugin-secrets-nats: plugin address: metadata=true network=unix address=/tmp/plugin454960758 timestamp=2023-04-21T08:30:48.796Z
2023-04-21T08:30:48.796Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: using plugin: metadata=true version=4
2023-04-21T08:30:48.859Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: setup: transport=gRPC status=started
2023-04-21T08:30:48.859Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats.stdio: waiting for stdio data: metadata=true
2023-04-21T08:30:48.862Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: setup: transport=gRPC status=finished err=<nil> took=3.425275ms
2023-04-21T08:30:48.862Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: special paths: transport=gRPC status=started
2023-04-21T08:30:48.863Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: special paths: transport=gRPC status=finished took="745.36µs"
2023-04-21T08:30:48.863Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: type: transport=gRPC status=started
2023-04-21T08:30:48.864Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: type: transport=gRPC status=finished took="488.678µs"
2023-04-21T08:30:48.864Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: version: transport=gRPC status=started
2023-04-21T08:30:48.864Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: version: transport=gRPC status=finished took="560.649µs"
2023-04-21T08:30:48.864Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: cleanup: transport=gRPC status=started
2023-04-21T08:30:48.865Z [TRACE] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: cleanup: transport=gRPC status=finished took="756.553µs"
2023-04-21T08:30:48.866Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats.stdio: received EOF, stopping recv loop: metadata=true err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-04-21T08:30:48.867Z [INFO] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: plugin process exited: metadata=true path=/etc/vault/vault_plugins/vault-plugin-secrets-nats pid=33
2023-04-21T08:30:48.867Z [DEBUG] secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_a9ca485e.vault-plugin-secrets-nats: plugin exited: metadata=true
2023-04-21T08:30:48.867Z [INFO] core: successfully mounted: type=vault-plugin-secrets-nats version="" path=nats-secrets/ namespace="ID: root. Path: "
```
**Expected behavior**
I'd expect, that i can successfully register the plugin on the first approach.
**Environment:**
* Vault Server Version (retrieve with `vault status`):
```
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.12.5
Build Date 2023-03-23T12:51:46Z
Storage Type file
Cluster Name vault-cluster-45b7d7a0
Cluster ID c3efee13-6b46-8025-03aa-0a9abeab41ca
HA Enabled false
```
* Vault CLI Version (retrieve with `vault version`): `Vault v1.12.3 (209b3dd99fe8ca320340d08c70cff5f620261f9b), built 2023-02-02T09:07:27Z`
I believe the essence of the problem, is that the Vault plugin mechanism is fundamentally incompatible with Kubernetes. It appears to be designed for deployments on VMs, where plugins are only updated whilst stable Vault servers continue to run undisturbed.
I proposed in the above-linked issue:
I wonder if HashiCorp would be willing to have a conversation about making the checksum verification of plugins optional … the current approach doesn’t seem well suited to maintaining uptime of a cluster during upgrade in K8s?
There was a response, but I declined to take the lead on pursuing it, as I personally do not run Vault on Kubernetes.