Help: Keyring has not been initialized yet

mclare · December 16, 2022, 6:49pm

Hi folks. Little help?

I had a v1.3.5 server run out of storage. During the hard restart, I mistakenly thought "Now’s a good chance to deploy v1.4.3.

Now I’m getting server log errors

Dec 16 13:46:33 pi4B-01 nomad[6936]: 2022-12-16T13:46:33.017-0500 [ERROR] worker: failed to submit plan for evaluation: worker_id=1cbbf7f3-stuff-f0329852a6be eval_id=6b45fd18-stuff-954392768f59 error="rpc error: keyring has not been initialized ye>
Dec 16 13:46:33 pi4B-01 nomad[6936]: 2022-12-16T13:46:33.017-0500 [ERROR] worker: error invoking scheduler: worker_id=1cbbf7f3-stuff-f0329852a6be error=“failed to process evaluation: rpc error: keyring has not been initialized yet”

Removing the data_dir and providing a peers.json have not worked, and I’m out of tricks. Help please.

shantanugadgil · December 16, 2022, 6:52pm

maybe related to:

github.com/hashicorp/nomad

Not upgrading servers across all regions to 1.4 causes "keyring has not been initialized yet"

opened 01:33PM - 15 Oct 22 UTC

closed 05:21PM - 17 Oct 22 UTC

optiz0r

type/bug theme/keyring

### Nomad version Main region: 1.4.1 Alternate region: 1.3.3 ### Issue … After upgrading 3x servers in the main region (which included the leader node) from 1.3.5 to 1.4.1, it was impossible to schedule any new jobs until the a second region containing a single server was also upgraded from 1.3.3 to 1.4.1. All evals failed with error "keyring has not been initialized yet". As soon as the alternate region server was upgraded, jobs in the main region started to be allocated once more. ### Reproduction steps * Upgrade region containing leader form 1.3.5 to 1.4.1, leaving a server in a second region at 1.3.3. * Attempt to schedule any new jobs #### Expected Result Jobs to continue to be scheduled in the main cluster once all servers had been upgraded to 1.4.1 #### Actual Result Cluster was in-operable until all servers across all regions were upgraded. #### Possible remedies The upgrade-specific instructions hint that there could be problems if the leader is allowed to rollback to an older cluster version, but not that servers across all regions need to be upgraded simultaneously in order to maintain cluster function. Either the upgrade nodes should call out this requirement, or the leader should be able to handle servers in another region still running on an older version. Typically I would only upgrade one region at a time. ### Nomad Server logs ``` Oct 14 21:58:31 leader.example.com nomad[276724]: 2022-10-14T21:58:31.265Z [ERROR] worker: error invoking scheduler: worker_id=4ba20503-dafb-ee08-c327-7e0d1779e4ae error="failed to process evaluation: rpc error: keyring has not been initialized yet" Oct 14 21:58:31 leader.example.com nomad[276724]: 2022-10-14T21:58:31.725Z [ERROR] worker: failed to submit plan for evaluation: worker_id=4ba20503-dafb-ee08-c327-7e0d1779e4ae eval_id=6c8ce736-2a39-836b-1d83-e51714187792 error="rpc error: keyring has not been initialized yet" Oct 14 21:58:31 leader.example.com nomad[276724]: 2022-10-14T21:58:31.726Z [ERROR] worker: error invoking scheduler: worker_id=4ba20503-dafb-ee08-c327-7e0d1779e4ae error="failed to process evaluation: rpc error: keyring has not been initialized yet" Oct 14 21:58:32 leader.example.com nomad[276724]: 2022-10-14T21:58:32.732Z [ERROR] worker: failed to submit plan for evaluation: worker_id=4ba20503-dafb-ee08-c327-7e0d1779e4ae eval_id=6c8ce736-2a39-836b-1d83-e51714187792 error="rpc error: keyring has not been initialized yet" Oct 14 21:58:32 leader.example.com nomad[276724]: 2022-10-14T21:58:32.732Z [ERROR] worker: error invoking scheduler: worker_id=4ba20503-dafb-ee08-c327-7e0d1779e4ae error="failed to process evaluation: rpc error: keyring has not been initialized yet" ```

mclare · December 16, 2022, 8:02pm

Thanks. It healed itself in about an hour. Perhaps that was the timeout of the original client?