HCSEC-2021-12 - Codecov Security Event and HashiCorp GPG Key Exposure

Security
, , ,
1

Bulletin ID: HCSEC-2021-12
Publication Date: April 22, 2021

Summary
HashiCorp was impacted by a security incident with a third party (Codecov) that led to potential disclosure of sensitive information. As a result, the GPG key used for release signing and verification has been rotated. Customers who verify HashiCorp release signatures may need to update their process to use the new key.

Codecov Security Event
On April 15, 2021, Codecov (a code coverage solution) publicly disclosed a security event during which an unauthorized party was able to make modifications to a Codecov component that Codecov customers download and execute when using the solution.

These modifications allowed the unauthorized party to potentially export information stored in Codecov users’ continuous integration (CI) environments. Codecov disclosed that the unauthorized access began on January 31, 2021, and was identified/remediated on April 1, 2021.

The Codecov disclosure is available at https://about.codecov.io/security-update/.

On investigation, HashiCorp found that a subset of HashiCorp CI pipelines used the affected Codecov component.

Exposure of HashiCorp GPG Key
The GPG private key used for signing hashes used to validate HashiCorp product downloads (SHA256SUM files, as available from https://releases.hashicorp.com and documented at https://hashicorp.com/security) was exposed.

While investigation has not revealed evidence of unauthorized usage of the exposed GPG key, it has been rotated in order to maintain a trusted signing mechanism. A new GPG keypair (fingerprint C874 011F 0AB4 0511 0D02 1055 3436 5D94 72D7 468F) has been published, and the exposed GPG keypair (fingerprint 91A6 E7F8 5D05 C656 30BE F189 5185 2D87 348F FC4C) has been revoked. Existing releases have been validated and re-signed, and updated information regarding key status and signature verification published to https://hashicorp.com/security.

Note that this exposure only affects HashiCorp’s SHA256SUM signing mechanism. MacOS code signing/notarization and Windows AuthentiCode signing of HashiCorp releases for those platforms were unaffected by the exposed GPG key in question. Signing for Linux packages (Debian and RPM) available from releases.hashicorp.com was also unaffected.

There is Terraform-specific context to consider. Terraform automatically downloads provider binaries during the terraform init operation and performs signature verification during this process. HashiCorp has published patch releases of Terraform and related tooling which update the automatic verification code to use the new GPG key, and provided separate Terraform-specific guidance.

Exposure of Other Information
HashiCorp has performed additional remediations related to information potentially exposed during this incident. Incident response activities are ongoing, and relevant updates and outcomes will be shared promptly when available via https://discuss.hashicorp.com/c/security.

HashiCorp Customer Impact
In general, customers should ensure that they download HashiCorp products only from the official release channel accessible directly at https://releases.hashicorp.com or linked from HashiCorp web properties.

In customer environments where HashiCorp product downloads are manually or automatically validated using the SHA256SUM files and associated signatures, process or configuration updates may be necessary to reflect the change in HashiCorp’s GPG key.

We deeply appreciate any effort to discover and disclose security vulnerabilities responsibly. For information about security at HashiCorp and the reporting of security vulnerabilities, please see https://hashicorp.com/security.

10 Likes
2

Frequently Asked Questions (added April 27 2021)

Incident response activities are ongoing, and relevant updates and outcomes will be shared promptly when available via https://discuss.hashicorp.com/c/security.

Has any HashiCorp customer data been disclosed?

There is no evidence of HashiCorp customer data disclosure at this point in time.

Was HashiCorp source code and/or binaries maliciously modified?

There is no evidence of malicious modification to HashiCorp code or binaries at this point in time.

What steps should HashiCorp customers/users consider taking?

In general, HashiCorp customers/users should ensure that they download HashiCorp products only from the official release channel accessible directly at https://releases.hashicorp.com or linked from HashiCorp web properties.

In environments where HashiCorp product downloads are manually or automatically validated using the SHA256SUM files and associated signatures, process or configuration updates may be necessary to reflect the change in HashiCorp’s GPG key.

HashiCorp has provided separate Terraform-specific guidance. Customers should consider upgrading to Terraform v0.11.15, v0.12.31, v0.13.7, v0.14.11, and v0.15.1 which have been released and use the new GPG key for provider validation.

What exactly was HashiCorp’s exposure?

The Codecov Bash Uploader and associated affected components as described in the Codecov disclosure was enabled for a small subset of HashiCorp-owned source code repositories and associated build pipelines.

Per the Codecov disclosure, the unauthorized alterations to their Bash Uploader enabled a third party to potentially export information stored in their users’ continuous integration (CI) environments. This information (specifically, repository names/locations and environment variables) potentially could have been sent to a third-party server outside of Codecov’s infrastructure.

On review of the CI environments for the affected HashiCorp repositories, a number of environment variables containing sensitive secrets (including the HashiCorp GPG private key used for signing release hashes) were determined to be potentially exposed.

How have HashiCorp sources, builds, releases, and/or binaries been verified?

The immediate focus has been on verification of existing builds, releases, and binaries. Analysis of logs, signatures, and storage system metadata in conjunction with comparison to known-good copies did not uncover evidence of malicious modification.

The secondary focus has been on verification of source code. Activity around HashiCorp source code repositories for the window of exposure has been and will continue to be reviewed.

There is no evidence of malicious modification to HashiCorp code or binaries at this point in time.

What was the timeline?

The Codecov disclosure was posted on April 15, 2021 and stated unauthorized third-party access occurred between January 31, 2021 and April 1, 2021.

HashiCorp began response immediately on review of the Codecov disclosure on April 15, 2021. Various investigative and remedial activities have been undertaken and continue.

HashiCorp rotated and revoked the exposed GPG key, re-signed the majority of existing product releases with the new GPG key, and published a public security bulletin on April 22, 2021.

HashiCorp released updated Terraform binaries with updated GPG keys on April 26, 2021.

3

Status Update (added May 4, 2021)

Initial response and investigation activities associated with this security event have been completed.

There was no evidence of HashiCorp customer data disclosure and no evidence of malicious modification to HashiCorp source code or binaries.

Efforts to prevent or minimize exposure to similar events in future are ongoing.