RPM Repository not available

zicklam · January 5, 2022, 2:06pm

We try to mirror the rpm repository: https://rpm.releases.hashicorp.com/RHEL/7/x86_64/stable/, but it fails.
Our company’s security does not allow to contact an rpm mirror directly, we need to mirror the full repo internally.

Could you please fix this repository, so we can sync again?

Thank you very much

blake · January 6, 2022, 6:18am

Hi @zicklam,

Would you mind sharing additional info as to how you are attempting to mirror this repository, and the specific error you are encountering?

Thanks.

zicklam · January 6, 2022, 8:40am

Hi @blake ,

we are using Pulp (v2.15.3) and feeding our repository from the URL:

Description:          upstream-shared7-hashicorp feed by
                      https://rpm.releases.hashicorp.com/RHEL/7/x86_64/stable

We are getting following error message on sync:

+----------------------------------------------------------------------+
         Synchronizing Repository [upstream-shared7-hashicorp]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[\]
... completed

Downloading repository content...
[-]

Task Failed

Malformed repository: metadata is missing for some packages in filelists.xml and
in other.xml

When I visit the feed URL with an standard browser, i’ll get 404 Not Found message:

https://rpm.releases.hashicorp.com/RHEL/7/x86_64/stable

blake · January 7, 2022, 8:08am

Thanks for the info, @zicklam.

I tried reproducing this issue using Pulp 3.17.2. I ran into a different error, but was able to resolve it and begin mirroring the repository.

I followed the instructions at https://pulpproject.org/pulp-in-one-container/ to set up Pulp. The only change I made was adding the following ALLOWED_CONTENT_CHECKSUMS line to settings.py.

ALLOWED_CONTENT_CHECKSUMS = ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']

After starting Pulp and setting an admin password (per the setup instructions), and installing the pulp CLI, I proceeded with creating the config to mirror the repo.

$ pulp rpm remote create \
    --name hashicorp \
    --url https://rpm.releases.hashicorp.com/RHEL/7/x86_64/stable \
    --policy on_demand

$ pulp rpm repository create \
    --name upstream-shared7-hashicorp \
    --remote hashicorp \
    --retain-package-versions 1 \ 
    --gpgcheck 1

I then tried to sync the repo using pulp rpm repository sync --name upstream-shared7-hashicorp, but received an error.

Started background task /pulp/api/v3/tasks/cc1a4c2a-902e-4711-84e1-bca95b79c224/
…Error: Task /pulp/api/v3/tasks/cc1a4c2a-902e-4711-84e1-bca95b79c224/ failed: ‘Content at the url https://rpm.releases.hashicorp.com/RHEL/7/x86_64/stable/repodata/148a0b63ab1f25fbcf16638c9462126b9cc17da7-other.xml.gz does not contain at least one trusted hasher which is specified in ‘ALLOWED_CONTENT_CHECKSUMS’ setting.’

This error is notably different than what you received, but the end result was the same; I could not synchronize the repo.

I came across the bug report Issue #9362: Mirroring rpm repositories with sha1 metatdata fails - RPM Support - Pulp where the reporter mentions that…

The metadata files of those repositories are mentioning sha instead of sha1 , pulp tries to mirror the repositories by setting this to sha, but that’s not a valid option in the repository object.

The solution was to explicitly set metadata_checksum_type and package_checksum_type on the repository to sha1.

$ pulp rpm repository update \
    --name upstream-shared7-hashicorp \
    --package-checksum-type sha1 \
    --metadata-checksum-type sha1

After this, the sync was successfully able to run.

$ pulp rpm repository sync --name upstream-shared7-hashicorp                                                                 
Started background task /pulp/api/v3/tasks/9a57e994-606f-4ab8-aa16-651c13c2ef9b/
.Done.

I’m not sure whether the same will work for you since you are using an older version of Pulp, but its worth a try to see if explicitly setting the hash type will resolve your issue.

You can find repo’s metadata file at https://rpm.releases.hashicorp.com/RHEL/7/x86_64/stable/repodata/repomd.xml. That file contains references to additional files under the repodata/ directory that contain the list of packages available from the repo, their versions, etc.

I hope this info helps. Let me know if you have any other questions and I will try to assist.

zicklam · February 3, 2022, 3:59pm

Hi, thank you for your help.

I’ve tried with a clean pulp3 instance and could sync the repository with the sha1 checksum settings.

But the packages can not be installed without deltarpm:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
consul-1.11.2.x86_64.rpm       FAILED                                          
https://pulp3/pulp/content/upstream-shared7-hashicorp/Packages/c/consul-1.11.2.x86_64.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=upstream-shared7-hashicorp clean metadata

What did changed on your repo since Consul Version 1.10.3? This is the last version we synced successfully with our pulp.

dralley · August 3, 2022, 3:06am

FYI, I work on Pulp, so I’m available to field questions @blake.

In any event I’d like to point out the underlying issues which caused Pulp to get confused:

[ERROR] Duplicate package 'consul-0:1.8.18-1.x86_64'
 appears 2 times with pkgids ['79b3b4204409fa046547f008a8c702f97a9e5a7f', '2ad326af7c51ba2414ed980414b5316d388253bd']
[ERROR] Duplicate package 'consul-0:1.10.4-1.x86_64'
 appears 2 times with pkgids ['6acf45a31663e12d3477cc7c31b4feb6b0482ed7', '88e6b8ba9aa3f807d43b7b82313570ffa16117c8']
[ERROR] Duplicate package 'consul-enterprise-0:1.11.0+ent-1.x86_64'
 appears 2 times with pkgids ['bf39f4b02821c502f855ed59afd8df3647204d7b', 'fc430d65fb9b8eade5e3f2f59d99a04e990cdb3e']
[ERROR] Duplicate package 'consul-0:1.11.0-1.x86_64'
 appears 2 times with pkgids ['1e4ac1e8666d57594e715a5400f624cbbd9eb891', '17fe0cbc9f6a938ed141b21d6d013a160b09edc3']
[WARNING] Insecure checksum type 'sha' is used for packages

Basically packages are supposed to be unique within the repository, but it appears that a few packages were rebuilt, renamed and then both the old and new packages (with the same version metadata) kept in the repository simultaneously. You’re not really supposed to do this - only one package should have a given “name-epoch-version-release-architecture” combination, and depending on the reason why there’s more than one it’s not clear which package will end up being installed as a result. In this situation DNF and Yum will pick the one with the most recent build time, but it’s kind of a fuzzy last-ditch heuristic rather than something to be relied on.

Older versions of Pulp didn’t deal with this very gracefully nor did it give clear error messages, which is something we’ve tried to work on - which has led me here, because users reported these repos as triggering the new error messages.

github.com/pulp/pulp_rpm

Error message (or warning) about ambiguous metadata during sync

opened 02:52PM - 14 Feb 22 UTC

closed 02:52PM - 14 Feb 22 UTC

dralley

You are probably here because you've seen a warning in your logs, or an error in… your sync task, with a message that looks like this: > The repository metadata being synced into Pulp is erroneous in a way that makes it ambiguous, and therefore we do not allow it to be synced in 'mirror_complete' mode. Please choose a sync policy which does not mirror repository metadata. OR > The repository metadata being synced into Pulp is erroneous in a way that makes it ambiguous. Yum, DNF and Pulp try to handle these problems, but unexpected things may happen. The reason you are seeing this warning (or error) is that the RPM repository metadata you are attempting to sync is not fully correct - it lists at least one package multiple different times, which makes it ambiguous which one should be selected when a client (such as Yum, DNF, or Pulp) try to download the package. When Yum or DNF encounters this, they will often just pick the first matching package they find without complaining, but this can sometimes lead to an undesired result. Pulp is more strict about this because the consequences are greater if it "chooses wrongly" when faced with this ambiguity - and in any case, Yum and DNF are not guaranteed to permit this scenario forever. It is "undefined behavior", so to speak. If you have seen this message, please file an issue here with the URL of the repository which you were syncing when this error / warning occurred, and we will attempt to reach out to the administrators of that repository to fix their metadata. If it is an internal or non-public repository, please contact the administrator responsible for that repository and direct them to this explanation. ----- If you are administering such a repo, and you are using the `createrepo` or `createrepo_c` tools, the most likely cause is one of the following: * Mutliple directories which share files (or contain symlinks to the files) were provided to the tool when the repository metadata was being generated - the package should only be present once. * A package was simply duplicated with a new filename - this does not change the internal metadata of the RPM package, so it is still considered the exact same package by most tooling * A package was built twice, or two identical packages were given different signatures, and different filenames - as in the previous case, even though they may be different files, the the key metadata (NEVRA) will be shared, and that creates ambiguity about which should be installed If you have questions, please reach out via our [Matrix](https://matrix.to/#/#pulp-rpm.:matrix.org?via=libera.chat&via=matrix.org) channel.

It would be great to see this resolved.

Also, using a stronger checksum than SHA1 is a good idea, as mentioned earlier - Pulp distrusts SHA1 by default so it removes a practical roadblock for some users in that sense, but also it’s just a sensible thing to do on its own merits as the cryptographic security of SHA1 is quite broken.

dralley · August 5, 2022, 4:19pm

@blake Ping, just in case you missed it ^^

gwgonza · August 25, 2022, 3:27pm

Has this been resolved? Seems like RHEL8 repository is still having issues with ambiguous metadata during sync.

dralley · August 25, 2022, 5:28pm

The repo isn’t fixed, no.

Newer versions of pulp (as in, starting about 2 weeks ago) added some crazy heuristics to be able to deal with these repos so that they are syncable and hopefully work without issue.

It would still be great to address the repository itself too though.

gwgonza · August 25, 2022, 6:01pm

@dralley It would be great to add address this at the repo itself. Do you know of any workaround for this at the moment? I’ve tried to sync content-only but still no luck.

dralley · August 25, 2022, 6:25pm

Which version are you using?

gwgonza · August 25, 2022, 6:28pm

I’m using pulp3 but I’m syncing to Satellite6 custom product not to a seprate pulp server.

dralley · August 25, 2022, 6:49pm

I meant which version of pulp 3 (specifically which version of the RPM plugin).

This should be addressed in pulp_rpm 3.17.10 (or later)

If you’re using the Katello RPMs, they’ll probably get the upgrade in a week or two.

gwgonza · August 25, 2022, 7:02pm

I will be waiting a week or two then. Thanks.

gwgonza · August 25, 2022, 7:52pm

python3-pulp-rpm-3.14.14-1
python3-pulpcore-3.14.16-1

dralley · August 25, 2022, 8:17pm

You probably need to upgrade Katello then, I’m not sure they’re going to pick up a release with that patch.

lsstreet · September 6, 2022, 5:34pm

Using Satellite version 6.10 and the version of pulp is:
python3-pulp-rpm-3.14.14-1.el7pc.noarch
python3-pulpcore-3.14.16-1.el7pc.noarch

I have the same issue on RHEL8 repository

zicklam · December 5, 2022, 9:31am

Pulpcore 3.21 with pulp-rpm 3.18.5 still not working on default setup.

Upvote for this

dralley · December 5, 2022, 9:37pm

Not working because of the checksum type, right?

zicklam · December 6, 2022, 11:24am

Yes. Why is hashicorp not fixing this issue on their side?

Topic		Replies	Views
Hashicorp repos down? HashiCorp Cloud Platform (HCP)	1	611	December 9, 2022
Hashcorp RPM repository seems like not alive Terraform	5	1326	August 25, 2022
Getting 404 on RHEL repo Vagrant	1	248	December 9, 2022
Adding Linux REPO to RedHat Satellite Terraform	3	2550	January 17, 2023
RPM stable repo is down Vault	0	327	April 25, 2023

RPM Repository not available

Related topics