Get AWS providers region in Golang

[samuel-phan]

Hi there!

I’m writting a Golang tool and I’d like to extract the region attribute from potentially several providers aws.

Eg:

provider "aws" {
  region = "us-east-1"
  alias  = "us-east-1"
}

provider "aws" {
  region = "us-west-2"
  alias  = "us-west-2"
}

resource "aws_dynamodb_table" "foo" {
  provider = aws.us-east-1

  name = "foo"

  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

 attribute {
    name = "LockID"
    type = "S"
  }
}

or:

provider "aws" {
  region = var.aws_region
}

variable "aws_region" {
  default = "us-east-1"
}

My tool takes for example the DynamoDB resource aws_dynamodb_table.foo and will make some calls to the AWS API using the AWS SDK for Golang v2, but then, I need the AWS region to make that AWS API call.

I’ve been reading about the packages:

• hclparse package - github.com/hashicorp/hcl/v2/hclparse - Go Packages
• hclsimple package - github.com/hashicorp/hcl/v2/hclsimple - Go Packages
• hclwrite package - github.com/hashicorp/hcl/v2/hclwrite - Go Packages
• gohcl package - github.com/hashicorp/hcl/v2/gohcl - Go Packages

but I’m kinda lost on which one I’d need.

My feeling is that I need to use gohcl package - github.com/hashicorp/hcl/v2/gohcl - Go Packages so I can add an hcl.EvalContext with some variables for the aws_region a bit like this:

	ctx := &hcl.EvalContext{
		Variables: map[string]cty.Value{},
		Functions: map[string]function.Function{},
	}

taken from go-wardley/hcl/hcl.go at master · DavidGamba/go-wardley · GitHub .

But before I dig too far, is there a simple way from terraform to get these information by any chance?

All I need is the map of:

"provider[\"registry.terraform.io/hashicorp/aws\"]" = "us-east-1"
"provider[\"registry.terraform.io/hashicorp/aws\"].my_alias" = "us-west-2"

where these keys are coming from the TF state file, eg:

  "resources": [
    {
      "mode": "managed",
      "type": "aws_dynamodb_table",
      "name": "foo",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"].my_aws",
...

Thanks!

[apparentlymart]

Hi @samuel-phan,

If you want to build something that is general though to work with any arbitrary Terraform configuration then unfortunately I think that will be quite a difficult goal to reach, because Terraform allows provider configurations to be dynamic – it isn’t always just constant values as in your example – and so even Terraform itself doesn’t necessarily know the final provider configuration until somewhere during the planning operation, where all of the dependencies were already resolved.

If you only need this to work for a few specific configurations that are under your control, I think you could make things a lot easier for yourself by factoring out the information you need elsewhere into a form that both Terraform and your separate tool can consume in the same way.

For example, you could put a JSON file alongside your .tf files and load the relevant information from that file as part of your Terraform configuration:

locals {
  aws_config = jsondecode(file("${path.module}/aws_config.json"))
}

provider "aws" {
  region = local.aws_config.default.region
}

You could then make your separate tool directly read aws_config.json, using the JSON library in your favorite programming language. That other tool doesn’t need to know anything about Terraform, and instead only needs to know about this simpler format that your Terraform configuration also understands.

[samuel-phan]

Hi @apparentlymart ,

Thanks for your reply and your continuous dedicated support on TF

I totally forgot to mention that I need this info for a TF plan, and a coworker just pointed out to me that these infos are part of the TF plan:

Internals: JSON Output Format | Terraform | HashiCorp Developer

For anybody facing the same, first we need to get the TF plan.json:

terraform plan -out=plan
terraform show -json plan > plan.json

Then for example in the TF plan.json:

"resources": [
  {
    "address": "aws_route53_zone.this",
    "mode": "resource",
    "type": "aws_route53_zone",
    "name": "this",
    "provider_name": "registry.terraform.io/hashicorp/aws",
  }
]
...
  "configuration": {
    "provider_config": {
      "aws": {
        "name": "aws",
        "full_name": "registry.terraform.io/hashicorp/aws",
        "version_constraint": "\u003e= 4.0.0",
        "expressions": {
          "region": {
            "constant_value": "us-east-1"
          }
        }
      },
      "aws.ap-southeast-1": {
        "name": "aws",
        "full_name": "registry.terraform.io/hashicorp/aws",
        "alias": "ap-southeast-1",
        "version_constraint": "\u003e= 4.0.0",
        "expressions": {
          "region": {
            "constant_value": "ap-southeast-1"
          }
        }
      },

Hope it’ll help other folks!
Thanks!

[apparentlymart]

Hi @samuel-phan,

Indeed, if you are able to create a plan before you do this work then that does make things a little easier.

One thing I’d caution is that the provider_config property you’ve found and shown in your snippet here is describing the static configuration before evaluation, and so in this particular case your region expression has a constant_value property, but if it were configured dynamically instead then constant_value would be absent and you’d have references there instead, enumerating which other objects the expression refers to.

Terraform doesn’t include the fully-evaluated provider configuration as part of a plan because provider configuration is considered to be “ephemeral” in the sense that it only exists for the duration of one operation: when you subsequently apply a plan, Terraform must re-evaluate the provider configuration from scratch because it’s common for credentials to vary between plan and apply, such as if someone is using JWT-based (“OpenID Connect”) authentication where each operation gets its own time-limited JWT.

[samuel-phan]

Thanks @apparentlymart , interesting! I’ll spend some time on playing more with this, and will update here, in case it can help anyone else.

– EDIT –

Just to confirm that @apparentlymart is right about the use of variables in the provider block, the TF plan looks like this:

    "configuration": {
        "provider_config": {
            "aws": {
                "name": "aws",
                "full_name": "registry.terraform.io/hashicorp/aws",
                "version_constraint": "\u003e= 4.0.0",
                "expressions": {
                    "region": {
                        "references": [
                            "var.aws_region"
                        ]
                    }
                }
            }
        },

Just for the fun, I’ve tried this also:

provider "aws" {
  region = "${var.continent}-${var.region}"
}

variable "continent" {
  default = "us"
}

variable "region" {
  default = "east-1"
}

and the TF plan looks like this:

    "configuration": {
        "provider_config": {
            "aws": {
                "name": "aws",
                "full_name": "registry.terraform.io/hashicorp/aws",
                "version_constraint": "\u003e= 4.0.0",
                "expressions": {
                    "region": {
                        "references": [
                            "var.continent",
                            "var.region"
                        ]
                    }
                }
            }
        },

[samuel-phan]

I share here our workaround in case it can help. I doesn’t work in 100% of use cases, but for most cases, it should.

Scenarios

1. For a given main.tf like this:

provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  region = "ap-southeast-1"
  alias  = "ap-southeast-1"
}

we have this in the TF plan:

"resources": [
  {
    "address": "aws_route53_zone.this",
    "mode": "resource",
    "type": "aws_route53_zone",
    "name": "this",
    "provider_name": "registry.terraform.io/hashicorp/aws", <-- Default provider
  }
]
...
  "configuration": {
    "provider_config": {
      "aws": {
        "name": "aws",
        "full_name": "registry.terraform.io/hashicorp/aws", <-- Default provider
        "version_constraint": "\u003e= 4.0.0",
        "expressions": {
          "region": {
            "constant_value": "us-east-1" <-- Constant "region" value
          }
        }
      },
      "aws.ap-southeast-1": {
        "name": "aws",
        "full_name": "registry.terraform.io/hashicorp/aws",
        "alias": "ap-southeast-1", <-- Alias of this provider
        "version_constraint": "\u003e= 4.0.0",
        "expressions": {
          "region": {
            "constant_value": "ap-southeast-1"
          }
        }
      },

2. And for a given main.tf like this:

provider "aws" {
  region = var.aws_region
}

variable "aws_region" {
  default = "us-east-1"
}

we have this in the TF plan:

    "variables": {
        "aws_region": {
            "value": "us-west-2" <-- Final value set by -var or -var-file
        }
    },
...
"configuration": {
        "provider_config": {
            "aws": {
                "name": "aws",
                "full_name": "registry.terraform.io/hashicorp/aws",
                "version_constraint": "\u003e= 4.0.0",
                "expressions": {
                    "region": {
                        "references": [
                            "var.aws_region" <-- References the var.aws_region
                        ]
                    }
                }
            }
        },

3. And for a given main.tf like this:

provider "aws" {
  region = "${var.continent}-${var.region}"
}

variable "continent" {
  default = "us"
}

variable "region" {
  default = "east-1"
}

we have this in the TF plan:

"configuration": {
        "provider_config": {
            "aws": {
                "name": "aws",
                "full_name": "registry.terraform.io/hashicorp/aws",
                "version_constraint": "\u003e= 4.0.0",
                "expressions": {
                    "region": {
                        "references": [
                            "var.continent", <-- References var.continent
                            "var.region"     <-- References var.region
                        ]
                    }
                }
            }
        },

Implementation

The suggested workaround implementation will support cases 1 and 2 , but NOT 3 (which is not a common case).

• Each resource has a provider_name attribute that matches a provider in the provider_config section.
• In the provider_config section, we can get the region:
  • If constant_value : just take the value as is.
  • If references :
    • If references = [ "var.aws_region" ] (only 1 value in the list): read the value from the variables section.
    • If references = [ ... ] (> 1 values in the list): not supported.

[apparentlymart]

Thanks for sharing this result!

One caveat I’d warn about is that having the region set to something that uses a single variable but doesn’t take its value directly will be indistinguishable from your second case.

For (contrived) example:

  region = coalesce(var.region, "us-east-1")

This expression would choose us-east-1 if the input variable were null. But the analysis you described would only show that this expression refers to var.region, without giving any indication about that fallback to a default value.

This is unfortunately just the limitations of static analysis of the configuration. Terraform can only report what it knows before the dynamic evaluation process starts, and even Terraform doesn’t know what coalesce means until it actually runs that function at runtime.

[samuel-phan]

Thanks for this clarification! It’s always good to know and to be warned.

This is not something we do in my company, so we kinda accept this limitation, at least for the time being.

I need to find a good balance between:

• the time spent to solve 90% of the situations
• and the time to solve the 10% left which would have a workaround (like setting the region = us-east-1 just to run my CLI, and then revert back to whatever expression they had). I know it’s not ideal, but time is money, right?