Minio for Private Terraform Module Storage

Terraform
1

Hi, I am looking to use minio as a replacement for s3 as a source for my custom terraform modules. I am publishing a small set of modules to a bucket on my local minio deployment but when I set the source in the module resource I am getting some strange behavior. I have looked through the documentation and I understand that s3 typically wants a region defined in the uri for the source. In my case I have set my minio region to us-east-1 for simplicity.

Here is the snippet of the terraform I am trying to initilize

module "container" {
  source   = "s3://minio.local/tfmodule/lxc/1.0.0/current.tgz"
}

I have also tried a variety of sources such as:

s3::https://minio.local/tfmodule/lxc/1.0.0/current.tgz
s3::https://s3.minio.local/tfmodule/lxc/1.0.0/current.tgz
s3::https://s3-us-east-1.minio.local/tfmodule/lxc/1.0.0/current.tgz

The error I am getting while running is:

Downloading s3://minio.local/tfmodule/lxc/1.0.0/current.tgz for container...
╷
│ Error: Failed to download module
│ 
│   on container.tf line 4:
│    4: module "container" {
│ 
│ Could not download module "container" (container.tf:4) source code from "s3://minio.local/tfmodule/lxc/1.0.0/current.tgz":
│ AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'
│       status code: 400, request id: W78F30J5E5GCEW85, host id: K6WjpqkgZFHVe1ueBPuGXOjD290fZv6y2btO5kKDPMwYF7fue2OtS2uuoMIbCFO1A7HPAfvVBlU=
╵

What is strange is that it seems to want the region to be set to eu-west-1. I have checked and I do not have any environment variables such as AWS_REGION set. I have even tried setting the aws and minio provider definitions to see if that might help, but its always the same error about the authorization header region.

provider "aws" {
  region                      = "us-east-1"
  skip_credentials_validation = true
  skip_requesting_account_id = true

  endpoints {
    s3 = "http://minio.local"
  }
}
provider "minio" {
  endpoint = "minio.local"
  region="us-east-1"
}

I stumbled across this post "gzip: invalid header" with a `.gz` artifact and its the only other one that seems kind of related to this issue, but I believe this is for nomad and not terraform.

Minio works great as a remote state backend but I am just trying to solve this last issue for pulling modules from the s3 bucket. Thanks :slight_smile:

2

I’m not sure what’s going on here but hopefully the following assorted context is useful in further debugging:

  • Terraform’s module installer implements S3 fetching using a now-deprecated old version of the AWS SDK.

    The AWS SDK often makes assumptions that are true for the real S3 but might not be true for clone implementations, particularly when using its default settings. For module installation in particular Terraform is relying heavily on the default settings, because (unlike the S3 backend) there isn’t a big configuration block to specify various override settings in.

  • Terraform actually delegates to an upstream library called go-getter to implement module installation, so you can find the implementation there:

    go-getter/get_s3.go at e70221100018573cdc74411c95c19b2a372f6728 · hashicorp/go-getter · GitHub

  • There’s an undocumented special query string argument aws_profile that seems to configure the AWS SDK to use settings from a profile in the shared configuration file, instead of using the SDK’s default auto-discovery behavior:

    go-getter/get_s3.go at e70221100018573cdc74411c95c19b2a372f6728 · hashicorp/go-getter · GitHub

    Therefore you might be able to get it to behave in a way more like what minio expects by configuring a profile in your credentials file and then adding ?aws_profile=NAME to the source address.

  • AFAIK that AuthorizationHeaderMalformed error is coming from your minio server rather than from Terraform or go-getter, so it seems like it’s minio that’s expecting the region to be eu-west-1. Maybe you can reconfigure minio to expect us-east-1 instead, or set AWS_REGION=eu-west-1, or (if you tried what I mentioned the previous point) specify eu-west-1 as the region in the profile you specified.

  • The go-getter implementation has no awareness of the hashicorp/aws provider or Terraform’s s3 backend, so directly configuring those cannot possibly change the behavior of module installation from S3.

    (Global settings outside of Terraform, such as environment variables or AWS configuration files, can potentially affect all of these different features at once.)

3

Thank you for all of your help and research. I was unsuccessful in getting the s3 protocol to work. I tried some different workaround methods but it seems like the main issue is the assumptions made in the go-getter function. If I get some time later ill look into reworking that method.

In the meantime the workaround that I was able to implement was using the https method. I did try https at one point but I kept getting authorization errors. That seems to stem from AWS authorization method needing an extra header. ( Authenticating Requests: Using the Authorization Header (AWS Signature Version 4) - Amazon Simple Storage Service )

I had to set the bucket I was using to public readonly for the modules paths to bypass the authentication layer. This is acceptable in my case since this is a local lab environment and the minio is running on my laptop.


module "container" {   
  source   = "https://minio.local/tfmodule/lxc/1.0.0/current.tgz" 
}