Context: I have gitlab runners which are executing terraform init command which is pulling all necessary terraform modules. Recently, I started hitting github throttling issues (60 calls to github api per hour). So I am trying to reconfigure my pipeline so it uses Github user’s private key.
Currently, I have the following in my pipeline but it still doesn’t seem to work and private key isn’t used to pull the terraform modules.
- GITHUB_SECRET=$(aws --region ${REGION} ssm get-parameters-by-path --path /github/umotifdev --with-decryption --query 'Parameters[*].{Name:Name,Value:Value}' --output json);
- PRIVATE_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/private_key").Value' | base64 -d);
- PUBLIC_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/public_key").Value' | base64 -d);
- mkdir -p ~/.ssh;
- echo "${PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa;
- chmod 700 ~/.ssh/id_rsa;
- eval $(ssh-agent -s);
- ssh-add ~/.ssh/id_rsa;
- ssh-keyscan -H 'github.com' >> ~/.ssh/known_hosts;
- ssh-keyscan github.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_host;
- echo -e "Host github.com\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config;
- echo ${PUBLIC_KEY} >> ~/.ssh/authorized_keys
The error I am seeing in my pipeline is something like (which is basically throttling from github):
Error: Failed to download module
Could not download module "vpc" (vpc.tf:17) source code from
"https://api.github.com/repos/terraform-aws-modules/terraform-aws-vpc/tarball/v2.21.0//*?archive=tar.gz":
bad response code: 403.
Can anyone suggest how I can fix the issue? I have a lot of projects which are pulling from terraform registry and it is important for me to use github service account with ssh key to increase the github api limit to 5000/per hour.
Unauthenticated users can only have 60 requests/hour. If you are using an authenticated user you can increase it to 5000. https://developer.github.com/v3/#increasing-the-unauthenticated-rate-limit-for-oauth-applications You need create a personal token of any user and replace GITHUB_TOKEN. I think you used circle-ci? I had the same issue with terraform I solved it this way
# check rate limit as unauthenticated user
curl -i -n https://api.github.com/rate_limit
echo "machine api.github.com login $GITHUB_USER_NAME password $GITHUB_TOKEN" > ~/.netrc
# check rate limit as authenticated user
curl -i -n https://api.github.com/rate_limit
for reference this is also how you download go modules from private repositories https://golang.org/doc/faq#git_https
Thanks for your response. I use gitlab but it is the same principal.
I tried .netrc way before and it doesn’t work. Any public module from terraform registry used inside terraform template won’t make use of that user. It is simply pulling it without using configured credentials.
can you post the output of curl commands from above?
Hi, so here is part of the pipeline which is checking the limit left for the github user that I am using. Credentials used are definitely correct. .netrc file also exists.
Here are the logs: (importantly you can see that X-RateLimit-Remaining only changes by 1 which means that terraform which is pulling public modules from terraform registry is not using defined credentials).
$ echo -e "machine api.github.com login XXXXX password XXXXXX" > ~/.netrc;
$ curl -si https://api.github.com -u XXXXX:XXXXX |grep -m1 X-RateLimit-Remaining
X-RateLimit-Remaining: 4988
$ cd ${TF_DIR:-.}
$ rm -rf .terraform
$ terraform --version
Terraform v0.12.23
Your version of Terraform is out of date! The latest version
is 0.12.24. You can update by downloading from https://www.terraform.io/downloads.html
$ if [[ -z ${BACKEND_FILE} ]]; then # collapsed multi-line command
2020/03/25 08:58:38 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/03/25 08:58:38 [INFO] Terraform version: 0.12.23
2020/03/25 08:58:38 [INFO] Go runtime version: go1.12.13
2020/03/25 08:58:38 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init", "-backend-config=backend_configs/test.hcl", "-reconfigure"}
2020/03/25 08:58:38 [DEBUG] Attempting to open CLI config file: /builds/umotif/devops/eks-global-platform/.terraformrc
2020/03/25 08:58:38 Loading CLI configuration from /builds/umotif/devops/eks-global-platform/.terraformrc
2020/03/25 08:58:38 [INFO] CLI command args: []string{"init", "-backend-config=backend_configs/test.hcl", "-reconfigure"}
2020/03/25 08:58:38 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/03/25 08:58:38 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
Initializing modules...
2020/03/25 08:58:38 [DEBUG] Module installer: begin bastion
2020/03/25 08:58:38 [DEBUG] bastion listing available versions of umotif-public/bastion/aws at registry.terraform.io
2020/03/25 08:58:38 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/03/25 08:58:38 [DEBUG] fetching module versions from "https://registry.terraform.io/v1/modules/umotif-public/bastion/aws/versions"
2020/03/25 08:58:38 [DEBUG] GET https://registry.terraform.io/v1/modules/umotif-public/bastion/aws/versions
2020/03/25 08:58:39 [DEBUG] found available version "1.0.3" for umotif-public/bastion/aws
2020/03/25 08:58:39 [DEBUG] found available version "1.0.2" for umotif-public/bastion/aws
2020/03/25 08:58:39 [DEBUG] found available version "1.0.1" for umotif-public/bastion/aws
2020/03/25 08:58:39 [DEBUG] found available version "1.0.0" for umotif-public/bastion/aws
2020/03/25 08:58:39 [DEBUG] looking up module location from "https://registry.terraform.io/v1/modules/umotif-public/bastion/aws/1.0.3/download"
Downloading umotif-public/bastion/aws 1.0.3 for bastion...
2020/03/25 08:58:39 [DEBUG] GET https://registry.terraform.io/v1/modules/umotif-public/bastion/aws/1.0.3/download
2020/03/25 08:58:39 [DEBUG] will download "https://api.github.com/repos/umotif-public/terraform-aws-bastion/tarball/1.0.3?archive=tar.gz" to .terraform/modules/bastion
2020/03/25 08:59:30 [DEBUG] Module installer: bastion installed at .terraform/modules/bastion/umotif-public-terraform-aws-bastion-7869622
2020/03/25 08:59:30 [DEBUG] Module installer: begin eks-node-group-a
2020/03/25 08:59:30 [DEBUG] eks-node-group-a listing available versions of umotif-public/eks-node-group/aws at registry.terraform.io
2020/03/25 08:59:30 [DEBUG] fetching module versions from "https://registry.terraform.io/v1/modules/umotif-public/eks-node-group/aws/versions"
2020/03/25 08:59:30 [DEBUG] GET https://registry.terraform.io/v1/modules/umotif-public/eks-node-group/aws/versions
- bastion in .terraform/modules/bastion/umotif-public-terraform-aws-bastion-7869622
2020/03/25 08:59:30 [DEBUG] found available version "1.0.3" for umotif-public/eks-node-group/aws
2020/03/25 08:59:30 [DEBUG] found available version "1.0.2" for umotif-public/eks-node-group/aws
2020/03/25 08:59:30 [DEBUG] found available version "1.0.1" for umotif-public/eks-node-group/aws
2020/03/25 08:59:30 [DEBUG] found available version "1.0.0" for umotif-public/eks-node-group/aws
2020/03/25 08:59:30 [DEBUG] looking up module location from "https://registry.terraform.io/v1/modules/umotif-public/eks-node-group/aws/1.0.3/download"
2020/03/25 08:59:30 [DEBUG] GET https://registry.terraform.io/v1/modules/umotif-public/eks-node-group/aws/1.0.3/download
Downloading umotif-public/eks-node-group/aws 1.0.3 for eks-node-group-a...
2020/03/25 08:59:30 [DEBUG] will download "https://api.github.com/repos/umotif-public/terraform-aws-eks-node-group/tarball/1.0.3?archive=tar.gz" to .terraform/modules/eks-node-group-a
2020/03/25 08:59:31 [DEBUG] Module installer: eks-node-group-a installed at .terraform/modules/eks-node-group-a/umotif-public-terraform-aws-eks-node-group-3c711d6
2020/03/25 08:59:31 [DEBUG] Module installer: begin eks-node-group-b
2020/03/25 08:59:31 [DEBUG] will download "https://api.github.com/repos/umotif-public/terraform-aws-eks-node-group/tarball/1.0.3?archive=tar.gz" to .terraform/modules/eks-node-group-b
- eks-node-group-a in .terraform/modules/eks-node-group-a/umotif-public-terraform-aws-eks-node-group-3c711d6
Downloading umotif-public/eks-node-group/aws 1.0.3 for eks-node-group-b...
2020/03/25 08:59:31 [DEBUG] Module installer: eks-node-group-b installed at .terraform/modules/eks-node-group-b/umotif-public-terraform-aws-eks-node-group-3c711d6
2020/03/25 08:59:31 [DEBUG] Module installer: begin eks-node-group-c
2020/03/25 08:59:31 [DEBUG] will download "https://api.github.com/repos/umotif-public/terraform-aws-eks-node-group/tarball/1.0.3?archive=tar.gz" to .terraform/modules/eks-node-group-c
- eks-node-group-b in .terraform/modules/eks-node-group-b/umotif-public-terraform-aws-eks-node-group-3c711d6
Downloading umotif-public/eks-node-group/aws 1.0.3 for eks-node-group-c...
2020/03/25 08:59:31 [DEBUG] Module installer: eks-node-group-c installed at .terraform/modules/eks-node-group-c/umotif-public-terraform-aws-eks-node-group-3c711d6
2020/03/25 08:59:31 [DEBUG] Module installer: begin kms-eks
2020/03/25 08:59:31 [DEBUG] kms-eks listing available versions of umotif-public/kms/aws at registry.terraform.io
2020/03/25 08:59:31 [DEBUG] fetching module versions from "https://registry.terraform.io/v1/modules/umotif-public/kms/aws/versions"
2020/03/25 08:59:31 [DEBUG] GET https://registry.terraform.io/v1/modules/umotif-public/kms/aws/versions
- eks-node-group-c in .terraform/modules/eks-node-group-c/umotif-public-terraform-aws-eks-node-group-3c711d6
Downloading umotif-public/kms/aws 1.0.0 for kms-eks...
2020/03/25 08:59:31 [DEBUG] found available version "1.0.0" for umotif-public/kms/aws
2020/03/25 08:59:31 [DEBUG] looking up module location from "https://registry.terraform.io/v1/modules/umotif-public/kms/aws/1.0.0/download"
2020/03/25 08:59:31 [DEBUG] GET https://registry.terraform.io/v1/modules/umotif-public/kms/aws/1.0.0/download
2020/03/25 08:59:31 [DEBUG] will download "https://api.github.com/repos/umotif-public/terraform-aws-kms/tarball/1.0.0?archive=tar.gz" to .terraform/modules/kms-eks
- kms-eks in .terraform/modules/kms-eks/umotif-public-terraform-aws-kms-8a8796e
2020/03/25 08:59:33 [DEBUG] Module installer: kms-eks installed at .terraform/modules/kms-eks/umotif-public-terraform-aws-kms-8a8796e
.
.
.
$ curl -si https://api.github.com -u XXXX:XXXXX |grep -m1 X-RateLimit-Remaining
X-RateLimit-Remaining: 4987