How to dynamically pass region to `providers` within module

Terraform
1

Hi All,

Here is my use case -

folder structure

  • providers.tf
  • root.tf
  • variables.tf
  • modules
    • create_security_group
      • aws_security_group.tf

I am trying to create security group for a set of vpcs, these vpcs are present in different aws regions.
Note : The actual use case involves creation of lots of resources in loop, giving a short example here.

providers.tf

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.11.0"
      configuration_aliases = [ aws.us-east-1, aws.us-east-2 ]
    }
  }

  provider "aws" {
     //aws creds
      region = "us-east-1"
      alias = "us-east-1"
  }
  provider "aws" {
     //aws creds
      region = "us-east-2"
      alias = "us-east-2"
  }
}

variables.tf

variable security_groups {
type = list(object({
name = string,
vpc = string
region = string
}))
default = [
  {
    "name": "security_group1",
    "vpc": "vpc1",
   "region": "us-east-1"
  },
  {
    "name": "security_group2",
    "vpc": "vpc2",
     "region": "us-east-2"
  }
]

}

root.tf

module "create_security_group" {
  source                 = "./modules/create_security_group"
  for_each             = { for sg in var.security_groups : sg.name => sg }
  vpc_id                 = lookup(each.value, "vpc")
 name                  = each.key
  providers = {
     aws = aws.us-east-1   
  }
}

aws_security_group.tf

//create security group
resource "aws_security_group" "security_group" {
  vpc_id                 = var.vpc_id
  name                   = var.name
}

As terraform is not allowing creation of providers dynamically. I am planning to create providers.tf using some script.

In root.tf , I am passing providers to create_security_group module in order to create some resources in a particular aws region.

I would like to know if there is any way in which, I could send the configuration alias dynamically based on region.

 providers = {
     aws = aws.us-east-1    // wanted to do aws.lookup(each.value, "region") or anything else to                                 
                                           // pass configuration alias dynamically
  }

Your help is highly appreciated.

2

Unfortunately, some of the choices you’re making here about how you want to structure your configuration, put you directly in conflict with various architectural choices made in AWS, Terraform, and terraform-provider-aws:

  • AWS treats its regions as (mostly) independent clouds
  • terraform-provider-aws follows this by choosing to model region as a provider configuration setting rather than something overrideable at resource level
  • Terraform has some rather particular restrictions on how providers work with its overall dependency graph building and expression evaluation

Taken together, what you’re trying to do here just isn’t possible.

Fundamentally, Terraform requires that the assignment of providers to blocks is entirely statically declared, since this is part of figuring out the dependency graph between blocks, which happens before expressions are evaluated.

You did mention you simplified this configuration for this question - so it’s a little hard to tell whether some of the more unusual things in it are the results of the simplification, or not…

Do you really have a create_security_group module that only contains a single resource "aws_security_group" block? If so, why is this a module at all? It seems like unnecessary complexity.

Do you really have a security_groups variable, which just literally contains values which could have been written directly into literal resource "aws_security_group" blocks? Why is that, and where is the variable input data coming from in your production setup?

It’s hard to give good advice without more understanding of your requirements, but one possible option could be to have, in your root directory:

module "us-east-1" {
  source = "./modules/region"
  providers = {
    aws = aws.us-east-1
  } 

  security_groups = [for x in var.security_groups: x if x.region == "us-east-1"]
  # Add other kinds of things you need per-region here too
}

# Copy the above block out for as many regions as necessary
1 Like
3

Thank you for the quick response.
Yes, because of simplification, the code which I have shared here might not seem appropriate.

My use case was to create multiple instances of managed streaming kafka in AWS. Below are the steps involved in provisioning multiple kafka instances -

  1. Get details of the VPC in which MSK will be deployed.
  2. Get details of the subnets available in the VPC found in previous step.
  3. Create MSK cluster
  4. Create Secrets in AWS secrets manager for storing kafka user credential.
  5. Attach the secrets created in step 4 to the MSK cluster in step3.

All these steps 1 through 5 are present inside module and being called from root module. All these steps are being run inside a loop for creating each MSK instance.

The folder structure is same as given in the question. Instead of security group creation I am doing MSK creation. The input to this is-

kafka_input = [{
    "msk_cluster_name": "kafka1",
    "vpc": "vpcID",
    "no_of_brokers": 3,
    "broker_size": "kafka.t3.small"
  },
{
    "msk_cluster_name": "kafka2",
    "vpc": "vpcID2",
    "no_of_brokers": 3,
    "broker_size": "kafka.t3.small"
  }]

The values for this input are being taken from a seaparate file(e.g valueFile) which differs in dev, staging, prod.

This is currently deployed in production. Now we want to deploy kafka1 and kafka2 in two different regions of AWS.
The region details will be given in valueFile . This array might increase if someone wants to deploy kafka3 in region3.

Any thoughts on this ?

4

Basically what I proposed in my previous post, then. Renaming some things based on the additional info:

module "kafka_us-east-1" {
  source = "./modules/kafka"
  providers = {
    aws = aws.us-east-1
  } 
  for_each = {for x in var.kafka_input: x.msk_cluster_name => x if x.region == "us-east-1"}

  # Pass input to module as needed using each.value.key_name
}

# Copy the above block out for as many regions as necessary
5

Okay. Thank you for the response!