Nomad 0.11.4 Released

We are releasing Nomad 0.11.4 containing a backported bug fix that is critical for some users to successfully upgrade existing Nomad servers which rely on Vault. If you are already running Nomad 0.11.3 or later successfully, there is no need to upgrade. The upcoming Nomad 0.12.2 release will also contain this fix.

Nomad client agents are unaffected and do not need to be upgraded.

Revoking Vault Tokens

Prior to Nomad 0.11.2, a bug could prevent Nomad from properly revoking and purging Vault tokens. While these tokens would be safely revoked by Vault, Nomad would still have a record of them and continually try to remove them.

While Nomad 0.11.2 fixed this bug, users with a large number of old Vault tokens to revoke (hundreds of thousands to millions), could see their Nomad servers run out of memory or timeout after being elected leader due to a bug in batching revocation. Users affected by this bug will likely see the following in their leader server’s logs:

background token revocation errored

The following metric will likely stay at or near 0:

nomad.vault.distributed_tokens_revoked

Upon upgrading servers to v0.11.4 you can confirm you were affected by this bug and that it was fixed by looking for the new log line:

batching tokens to be revoked

That log line is only emitted when there are multiple batches of tokens to be revoked which should only occur when upgrading from pre-0.11.2 versions of Nomad. Monitoring the following metric should show the tokens being periodically revoked until the backlog of tokens to be revoked is processed:

nomad.vault.distributed_tokens_revoked

Go 1.14.6

Nomad 0.11.4 also upgrades to Go 1.14.6 from 1.14.3 to address two Go CVEs. We believe these CVEs are low severity for Nomad users and have only upgraded Go out of an abundance of caution. See issue 8441 for details.

The Nomad Team

Links:

Changelog - https://github.com/hashicorp/nomad/blob/v0.11.4/CHANGELOG.md

Binaries - https://releases.hashicorp.com/nomad/0.11.4/

2 Likes