One of the organizations I do work for prefers to install software from private repositories. For open-source projects, this has typically meant:
- Identify a public repository to pull contents from
- Identify best method to create a local/internal duplicate (mirror) of that public repository
- Point internal security-scanners against the local/internal mirror
- Allow internal clients to access only the contents that the security-scanners have “blessed”
For other projects, I’ve frequently been able to use (cron-enabled)
rsync job to replicate the upstream project-contents while minimizing the amount of duplicative fetches (i.e., no re-downloading content I already have locally). When I was trying to pull from the download site, I noticed that, if you gave an invalid URL specification, you’d get what looks like an S3 error message:
# 403 Forbidden * Code: AccessDenied * Message: Access Denied * RequestId: <REQIDSTRING> * HostId: <LONGALPHANUMSTRING>
WHich made me think, “ah ha! I can probably rclone this”. Unfortunately, I could neither figure out the requisite
rclone pull-config to do so, nor were my google searches terribly fruitful. Mostly, they pointed me at how to use the network provider rather than how to use rclone (or any tool, really) to set up a mirror.
As a kludge, I wrote a quick-n-dirty scraper BASH script to do things for me, but, it’s sub-ideal (“brittle” and not terribly avoiding of duplication).
So, figured I’d post here to see if anyone has done similar and could give me some tips.
Thanks in advance!