S3 Bucket Module Source "Failed to download module"

Terraform
1

Within the context of the command terraform init from an AWS CodeBuild instance of image hashicorp/terraform:0.12.19, the module fails to download.

The source terraform template references a module package source in a s3 bucket.

 module "outbound" {
   source        = "s3::https://s3.amazonaws.com/my-modules/outbound-0.0.1.zip"
...

The command fails with this output:

Error: Failed to download module

Could not download module "outbound" (main.tf:17) source code from
"s3::https://s3.amazonaws.com/my-modules/outbound-0.0.1.zip":
NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

This should be valid per the documentation:

The s3 endpoint is correct because the region is us-east-1.

The CodeBuild project has an IAM role with correct permissions to s3.

The code works from a local CLI command terraform init.

How do I get more information from the command output?

What am I doing wrong fetching the module package from s3?

I have tried many different s3 URLs.

The environment has the variable that would supply the task_role to terraform: ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/40292219-6c1c-424e-b072-8456e6ea8165

2

Hi @outthought,

Looking at the relevant code in the upstream library Terraform uses to fetch modules, it looks like it supports only three AWS credentials sources:

  • The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
  • The ~/.aws/credentials file (or equivalent on other platforms)
  • EC2 instance profile auth via the metadata service.

The Terraform AWS provider and S3 backend code checks for an additional environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and, if present, adds an HTTP credentials source that seems to be intended to work within ECS, based on the log line the provider/backend produces in that case:

[INFO] ECS container credentials detected, RemoteCredProvider added to auth chain

So it looks like in order for the module installer to support ECS-style authentication the upstream go-getter library would need to be adjusted to use logic similar to Terraform. Since go-getter is a shared library common to several other applications, I imagine it would need some API changes to allow the calling application to customize the credentials configuration, and then maybe Terraform could configure it using the same implementation it uses for the S3 backend.

1 Like
3

To summarize, at present, terraform init inside of AWS CodeBuild will fail to download modules from an S3 module source.

Thank you for the insight.

There are two workarounds I will mention in case they are helpful to someone.

One, a public s3 bucket and object, along with http or https module source (not prefix s3::).

Two, a github module source with the AWS CodeBuild buildspec attribute git-credential-helper.

Here are the details of option Two:
Terraform template module source:

module "outbound" {
  source = "github.com/MyOrganization/outbound?ref=0.0.1"
...

buildspec.yml:

version: 0.2

env:
  git-credential-helper: yes