Bulletin ID: HCSEC-2020-01
Affected Products / Versions: Previous versions of Nomad and Nomad Enterprise; fixed in 0.10.3.
Publication Date: 29 January, 2020
Summary
A vulnerability was identified in Nomad and Nomad Enterprise (“Nomad”) such that unbounded resource usage, triggered by the establishment of many unauthenticated HTTP or RPC connections, may generate excessive load and/or crash the server. This vulnerability, CVE-2020-7218, affects all previous releases of Nomad and is fixed in the 0.10.3 release.
Background
Nomad utilizes HTTP for networked communication between Nomad cluster members and control/administration of Nomad servers.
Nomad clients are configured to communicate with their regional servers and communicate using remote procedure calls (RPC) to register themselves, send heartbeats for liveness, wait for new allocations, and update the status of allocations. These RPC requests/responses traverse the network between Nomad nodes using TCP.
Nomad’s HTTP and RPC both support using end-to-end TLS with optional client authentication.
Details
An internal security review identified a vulnerability such that it was possible to consume excessive Nomad server resources via HTTP and RPC services.
A Nomad server was exposed to denial of service attack by any party with network-level connectivity to that server. Authentication via mutual TLS is not required in order to launch an attack.
While successful exploitation of this vulnerability would not affect confidentiality or data integrity within a Nomad deployment, availability would be affected.
Remediation
Customers should upgrade to Nomad or Nomad Enterprise 0.10.3, or newer. Please refer to Upgrading Nomad for general guidance and version-specific upgrade notes.
Several new configuration options are introduced (rpc_max_conns_per_client, rpc_handshake_timeout, http_max_conns_per_client, https_handshake_timeout) with default values which restrict resource usage and reduce exposure to attack.
If upgrade is not possible, customers may consider other mitigation options:
-
Restrict connectivity to Nomad servers to trusted sources/networks only. TCP ports used by Nomad for HTTP and RPC default to 4646 and 4647.
-
Restrict resource usage on Nomad servers by enforcing network connection limits. For example,
iptables -A INPUT -p tcp --syn --dport 4646 -m connlimit --connlimit-above 100 -j REJECT --reject-with tcp-reset.
Enabling mutual TLS as a defense-in-depth mechanism for Nomad clusters, per documentation, remains strongly recommended.
Acknowledgement
This issue was identified by the HashiCorp security team.
We deeply appreciate any effort to discover and disclose security vulnerabilities responsibly. For information about security at HashiCorp and the reporting of security vulnerabilities, please see https://hashicorp.com/security.