Summary
We have a 3 node vault enterprise cluster.
We want to migrate from Shamir keys unseal to HSM unseal.
During seal migration process after stepping down the active node and re-election of the new active node we are getting an error
Dec 15 21:35:33 apvault vault[680480]: 2025-12-15T21:35:33.752+0100 [WARN] error encrypting with seal: seal=pkcs11
Dec 15 21:35:33 apvault vault[680480]: 2025-12-15T21:35:33.752+0100 [ERROR] failed to encrypt value using any seal wrappers
Dec 15 21:35:33 apvault vault[680480]: 2025-12-15T21:35:33.752+0100 [ERROR] core: seal migration error:
Dec 15 21:35:33 apvault vault[680480]: error=
Dec 15 21:35:33 apvault vault[680480]: | error setting new recovery key information: failed to encrypt keys for storage: failed to encrypt recovery key for storage: error seal wrapping value: encryption generated no results
Dec 15 21:35:33 apvault vault[680480]: | error decrypting using seal pkcs11: encryption key object handle not found
Dec 15 21:35:33 apvault vault[680480]:
Configuration details
HSM is YubiHSM2 with the following keys
yubihsm-shell --connector=http://xxx.xxx.xxx.xxx:xxxxx
connect
id: 0x0001, type: authentication-key, algo: aes128-yubico-authentication, sequence: 1, label: DEFAULT AUTHKEY CHANGE THIS ASAP
id: 0x0002, type: authentication-key, algo: aes128-yubico-authentication, sequence: 0, label: VAULT_PKCS11_RSA
id: 0x0064, type: asymmetric-key, algo: rsa2048, sequence: 0, label: label_rsa_vault_autounseal
HSM is accessible from all vault nodes. The HSM service is exposed via HTTP.
seal stanza in vault.hcl
seal "pkcs11" {
name="pkcs11"
priority="1"
lib = "/usr/lib64/pkcs11/yubihsm_pkcs11.so"
slot = "0"
key_label = "label_rsa_vault_autounseal"
hmac_key_label = "VAULT_PKCS11_RSA"
}
vault.env
YUBIHSM_PKCS11_CONF=/opt/yubihsm/yubihsm_pkcs11.conf
VAULT_HSM_PIN=0002<password>
yubihsm_pkcs11.conf
connector = http://xxx.xxx.xxx.xxx:xxxx
debug
libdebug
debug-file = /opt/yubihsm/yubihsm_pkcs11_debug
libtool-ltdl installed
OS: RHEL8
environment variables loaded to vault systemctl process
cat /proc/PID/environ
...
YUBIHSM_PKCS11_CONF=/opt/yubihsm/yubihsm_pkcs11.conf
VAULT_HSM_PIN=0002<password>
What is tripping us even more that we have another cluster with exact HSM and vault configuration & architecture and it works.
These are the capabilities of the keys in HSM. The exact key setup works on another cluster.
id: 0x0064, type: asymmetric-key, algorithm: rsa2048, label: “label_rsa_vault_autounseal”, length: 896, domains: 10, sequence: 0, origin: generated, capabilities: decrypt-oaep:exportable-under-wrap
id: 0x0002, type: authentication-key, algorithm: aes128-yubico-authentication, label: “VAULT_PKCS11_RSA”, length: 40, domains: 10, sequence: 0, origin: imported, capabilities: decrypt-oaep:delete-asymmetric-key:export-wrapped:generate-asymmetric-key:get-pseudo-random:import-wrapped:put-wrap-key, delegated_capabilities: decrypt-oaep:export-wrapped:exportable-under-wrap:import-wrapped
However in working cluster the vault is deployed on RHEL8 machines and YubiHSM is deployed on RHEL8 machine as well. So yubihsm2-sdk-2023-11-centos7-amd64.tar.gz is used on HSM VM and vault VMs
In problematic cluster the vault is deployed on RHEL8 machines and YubiHSM is on RHEL10 machine. The vault machines use yubihsm2-sdk-2023-11-centos7-amd64.tar.gz tools (yubihsm_pkcs11.so) and the HSM machines uses yubihsm2-sdk-2025-06-fedora42-amd64.tar.gz tools.