I’m trying to mirror the hashicorp apt repository with debmirror, but I get the following error:
[GNUPG:] BADSIG DA418C88A3219F7B HashiCorp Security (HashiCorp Package Signing) <security+packaging@hashicorp.com>
[GNUPG:] VERIFICATION_COMPLIANCE_MODE 23
gpgv: Signature made Wed Jan 4 15:38:21 2023 UTC
gpgv: using RSA key DA418C88A3219F7B
gpgv: BAD signature from "HashiCorp Security (HashiCorp Package Signing) <security+packaging@hashicorp.com>"
.temp/.tmp/dists/focal/InRelease signature does not verify.
I’ve tried manually downloading the InRelease file and checking the signature with gpg --verify, and sure enough, I get a similar error about a bad signature.
Note that this is not the same as an error caused by missing the public key for the signature.
Strangely, if I add the repo to my local /etc/apt/sources.list.d directory, I am able to update fine.
I’m afraid I’m not familiar with debmirror and so I can only really guess what’s going on here, but the fact that using the repository directly with apt is working for you makes me wonder if debmirror has its own separate set of trusted keys.
Our instructions for setting up the APT repository call for specifying the expected key directly in the relevant line of sources.list, like this:
deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com focal main
(assuming you previously placed the keyring at that path)
I think this means that debmirror could only take that key into account if it were using sources.list as a data source, but I would be surprised if that were how it works because I wouldn’t want to have to set up a repository as a source for my own computer just to generate a mirror of it.
I’m sorry if the above seems like the wrong direction; I’m just doing my best with the information I have. If you have more information that might help explain what’s going on here then please share it!
If you download the InRelease file and try to verify it with
gpg --verify InRelease
it fails.
Having investigated a bit, I noticed that:
The signature declares itself as a signature of a binary document, whereas all the other examples of InRelease files on my computer contain signatures of a canonical text document
If I manually detach the signature from the file, and cut out the message content to a separate file, the signature verifies the separated file OK!
Putting these together, my hunch is that the software generating this signature is skirting into an area of non-compliance with RFC 4880: OpenPGP Message Format in regards to not performing canonicalization of line endings to CRLF before calculating the signature on a text document.
I guess apt just happens to be using a PGP implementation that tolerates the dubious compliance, whereas command line gpg (and debmirror) does not.
Also re-reading this thread again later I can see that @tmccombs already ruled out the possibility of the key not being found at all, but I didn’t quite follow that on my initial read until looking more closely now, so sorry for pulling in the wrong direction earlier.
The behavior I see on my system is a little different, but it still fails:
$ gpg --verify InRelease
gpg: Signature made Wed 04 Jan 2023 07:38:21 AM PST
gpg: using RSA key DA418C88A3219F7B
gpg: BAD signature from "HashiCorp Security (HashiCorp Package Signing) <security+packaging@hashicorp.com>" [unknown]
I’m not familiar with RFC 4880 but from a quick skim of the section you quoted here I’m understanding that all of the line endings are supposed to be CRLF. If I look at the InRelease file I downloaded I see that the -----BEGIN… line ends with just LF, and the blank line after the Hash: field is also represented as a pair of naked LF without any CR.
The fact that even the boundary lines are incorrectly terminated makes it seem like it’s the overall PGP signed message that has the wrong line endings, rather than just the input cleartext, but converting the overall InRelease file to use CRLF throughout doesn’t make it valid either, so indeed it seems like the signature itself was probably calculated with the wrong line endings as you say.
The APT repository is not Terraform-specific and so it’s not something I have visibility into the details of, but I agree that the results here don’t seem to comply with the spec so I’ll figure out a suitable place to send this feedback so that the team responsible for the APT repository might investigate further and hopefully produce a spec-compliant InRelease file.
Ah, sorry, I should have been a bit more verbose in my message - this behaviour is not different, it’s exactly what I see too.
A little bit of extra clarification on the CRLF issue: I wouldn’t expect to see actual CRLF sequences in the files on disk at all (since they are Linux flavoured files using Linux conventions) - it’s just that, when using the -----BEGIN PGP SIGNED MESSAGE----- format, the RFC says the signature calculation should internally transform the message to CRLF before computing signatures - and that is the part that seems to not be happening here.
The APT repository is not Terraform-specific and so it’s not something I have visibility into the details of
Sorry about that. I wasn’t sure what section to put this under.
Thanks for finding more details on the issue @maxb. I wasn’t familiar enough with PGP/gpg to figure that out
I am curious why it doesn’t cause an issue with apt-get though, since I’m pretty sure apt uses gpg under the covers. I wonder if it is passing some option to gpg, or detaching the signature before verifying, or maybe it just isn’t using the InRelease file?
I’m not sure either but one possible explanation is that the repository includes two different signed artifacts:
InRelease is a combination of the message itself and a signature all in one file.
Release is a raw release manifest which is signed by the detached signature in Release.gpg.
I wonder if normal apt is using the Release/Release.gpg pair and so is working with the raw Release file and whatever line endings it naturally has, while debmirror is trying to use InRelease and getting into trouble because the message portion of that file has had its line endings changed as part of the signing process.
I’ve not confirmed whether that’s a correct theory but I am still working on figuring out where to send this feedback internally so that we can make sure that all of the signatures conform to the relevant RFC.
Note: There’s no [Ubuntu|apt|PPA|General] section to put this under, so I also would have selected Terraform, but I found this existing post.
This started happening with apt-get [dist/jammy] today as well:
Err:4 https://apt.releases.hashicorp.com jammy InRelease
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY AA16FCBCA621E701
…
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://apt.releases.hashicorp.com jammy InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY AA16FCBCA621E701
W: Failed to fetch https://apt.releases.hashicorp.com/dists/jammy/InRelease The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY AA16FCBCA621E701
W: Some index files failed to download. They have been ignored, or old ones used instead.
It appears the public key for the Ubuntu PPA has either expired or is no longer available, which happens with other PPAs from time to time.
What you described here wasn’t true at the time of this original discussion, but is true now: the original signing key for the apt repository is no longer valid and there’s now a new one to use instead.
Thank you for the prompt reply and the link to the instructions.
Re-adding the “Hashicorp Repo” to my sources list resolved the issue.
Before, I didn’t have the reference to the keyring within the source declaration and needed to download the file.
I used the apt-key del command to delete the old key. I also tried doing “curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -” again to see that the new key was indeed added, but still get the same error.
You’ve posted this question on a topic in the Terraform category, so it’s unlikely that folks familiar with Vault will be watching here. I suggest starting a new topic in the Vault category instead.
Since the issue that this topic was originally discussing was fixed long ago, I’m going to close this topic. If you are a future reader finding this after seeing similar symptoms with either Terraform or some other package in the apt.releases.hashicorp.com repository then please open a new topic about it in the most relevant category for whatever package you were trying to install.
