Hi everyone,
I’m encountering an issue when trying to deploy HashiCorp Vault using Google Cloud Storage (GCS) as the storage backend. I’ve successfully configured Vault to use GCS, and I can see in the logs that Vault starts up correctly and transitions to standby mode after being unsealed with a stored key. However, after a few seconds, Vault begins to throw errors when attempting to acquire a lock.
The specific error message is: “failed to decode lock: unexpected end of JSON input”. This seems to indicate an issue with decoding a lock in JSON format. I’ve reviewed the GCS configuration, checked network connectivity, and verified the integrity of the data stored in the GCS bucket, but the problem persists.
Has anyone experienced a similar issue when using GCS as Vault’s backend? Any suggestions on what else I could check or how to troubleshoot this problem?
Any help or guidance would be greatly appreciated. Thanks in advance for your time and assistance!
Best regards,
LOGS:
==> Vault server started! Log data will stream in below:
2024-03-19T09:29:15.713Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2024-03-19T09:29:15.714Z [DEBUG] storage.gcs: configuring backend
2024-03-19T09:29:15.714Z [DEBUG] storage.gcs: creating client
2024-03-19T09:29:15.715Z [DEBUG] storage.gcs: configuration: bucket=tio-nep-vault2-backend-svcs chunk_size=8388608 ha_enabled=true max_parallel=0
2024-03-19T09:29:15.715Z [DEBUG] storage.gcs: creating client
2024-03-19T09:29:15.966Z [DEBUG] service_registration.kubernetes: "namespace": "vault"
2024-03-19T09:29:15.966Z [DEBUG] service_registration.kubernetes: "pod_name": "vault-0"
2024-03-19T09:29:16.179Z [DEBUG] core: set config: sanitized config="{\"api_addr\":\"\",\"cache_size\":0,\"cluster_addr\":\"\",\"cluster_cipher_suites\":\"\",\"cluster_name\":\"\",\"default_lease_ttl\":360000000,\"default_max_request_duration\":0,\"disable_cache\":false,\"disable_clustering\":false,\"disable_indexing\":false,\"disable_mlock\":true,\"disable_performance_standby\":false,\"disable_printable_check\":false,\"disable_sealwrap\":false,\"disable_sentinel_trace\":false,\"enable_response_header_hostname\":false,\"enable_response_header_raft_node_id\":false,\"enable_ui\":false,\"listeners\":[{\"config\":{\"address\":\"0.0.0.0:8200\",\"cluster_address\":\"0.0.0.0:8201\",\"tls_disable\":1},\"type\":\"tcp\"}],\"log_format\":\"standard\",\"log_level\":\"\",\"log_requests_level\":\"\",\"max_lease_ttl\":360000000,\"pid_file\":\"\",\"plugin_directory\":\"\",\"plugin_file_permissions\":0,\"plugin_file_uid\":0,\"raw_storage_endpoint\":false,\"seals\":[{\"disabled\":false,\"type\":\"gcpckms\"}],\"service_registration\":{\"type\":\"kubernetes\"},\"storage\":{\"cluster_addr\":\"\",\"disable_clustering\":false,\"redirect_addr\":\"\",\"type\":\"gcs\"}}"
2024-03-19T09:29:16.179Z [DEBUG] storage.cache: creating LRU cache: size=0
2024-03-19T09:29:16.344Z [INFO] core: Initializing version history cache for core
2024-03-19T09:29:16.345Z [DEBUG] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
2024-03-19T09:29:16.346Z [INFO] core: stored unseal keys supported, attempting fetch
2024-03-19T09:29:16.389Z [DEBUG] would have sent systemd notification (systemd not present): notification=READY=1
2024-03-19T09:29:16.552Z [DEBUG] core: starting cluster listeners
2024-03-19T09:29:16.553Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=0.0.0.0:8201
2024-03-19T09:29:16.553Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2024-03-19T09:29:16.553Z [INFO] core: vault is unsealed
2024-03-19T09:29:16.553Z [INFO] core: entering standby mode
2024-03-19T09:29:16.619Z [INFO] core: unsealed with stored key
2024-03-19T09:29:21.594Z [ERROR] core: failed to acquire lock: error="lock: attempt lock: write lock: failed to decode lock: unexpected end of JSON input"
2024-03-19T09:29:36.633Z [ERROR] core: failed to acquire lock: error="lock: attempt lock: write lock: failed to decode lock: unexpected end of JSON input"