Thanks for the extra details, @dipendra.chaudhary.
You mentioned at the end of your comment that your goal is to make this a reusable module for declaring security groups and their rules, so I want to start from that requirement and work backwards to discuss ways to achieve that. A Terraform module typically takes the data it will use directly as input variables, rather than indirectly via files on disk, and so I would typically start by defining what data this module will need as input variables:
variable "vpc_id" {
type = string
}
variable "default_security_group" {
type = object({
manage_ingress = bool
manage_egress = bool
})
# This variable is optional. Setting it
# means that the caller wants the module
# to manage the VPC's default security
# group.
default = null
}
variable "security_groups" {
type = map(object({
name = string
tags = map(string)
# AWS allows both rules that are matched by
# source network address and rules matched
# by source security group. They each need
# different handling by our module so we'll
# use two different attributes.
cidr_rules = map(object({
type = string
protocol = string
from_port = number
to_port = number
cidr_blocks = set(string)
description = string
}))
source_group_rules = map(object({
type = string
protocol = string
from_port = number
to_port = number
source_group_key = string
description = string
}))
}))
default = {}
}
Above I’ve defined three variables, the third of which is the main one that will take all of the data related to the security groups and their rules. It’s a two-level hierarchical structure because from your example it seemed like you wanted to have different rules for each group.
Let’s first write out the resource blocks for the default security group, since that isn’t changed much from what you had in your example. I made some changes here you didn’t actually call for just because I thought it useful to show the pattern of deciding whether to activate a particular feature based on whether a variable is null, which is pretty common in shared modules where the optional feature also needs some customizable parameters:
resource "aws_default_security_group" "this" {
# The [*] operator converts a non-list value
# that might be null into a list of either
# zero or one elements.
count = length(var.default_security_group[*])
vpc_id = var.vpc_id
}
resource "aws_security_group_rule" "default_ingress" {
# We'll filter out a non-null object if it
# doesn't also specify to manage ingress.
count = length([
for o in var.default_security_group[*] : o
if o.manage_ingress
])
security_group_id = aws_default_security_group.this[count.index]
source_security_group_id = aws_default_security_group.this[count.index]
type = "ingress"
protocol = -1
from_port = 0
to_port = 0
}
resource "aws_security_group_rule" "default_egress" {
count = length([
for o in var.default_security_group[*] : o
if o.manage_egress
])
security_group_id = aws_default_security_group.this[count.index]
cidr_blocks = ["0.0.0.0/0"]
type = "egress"
protocol = -1
from_port = 0
to_port = 0
}
With those defined we can now move on to the part which I think you are most interested in: defining the additional security groups and their associated rules. I designed the input variables above to use maps rather than lists because that allows us to use resource for_each to define the multiple objects in a way that will allow you to add and remove security groups and rules in future without disturbing other groups/rules. (There’s some more detail on this decision in When to use for_each instead of count.)
To meet the requirements for for_each we need a map that has one element for each instance we want to declare. var.security_groups is already of a suitable shape for the security groups themselves, but in order to declare the security group rules systematically with for_each we will need a collection with one element per rule across all security groups, and so we can use the flatten function in a local value to derive such a structure:
locals {
cidr_rules = flatten([
for group_key, group in var.security_groups : [
for rule_key, rule in group.cidr_rules : {
rule_key = rule_key
group_key = group_key
rule = rule
group = group
}
]
])
source_group_rules = flatten([
for group_key, group in var.security_groups : [
for rule_key, rule in group.source_group_rules : {
rule_key = rule_key
group_key = group_key
rule = rule
group = group
}
]
])
}
Each of these is a list of objects with rule, rule_key, group, and group_key attributes that describe all of the rules across all of the groups. We need to include the keys in there as well as the groups because the other requirement for for_each is that we can construct a unique key for every element, which in this case will be a combination of the group and rule keys, as we’ll see in a moment.
Now we can declare all of the groups and all of the rules as just three resources with for_each set:
resource "aws_security_group" "additional" {
for_each = var.security_groups
vpc_id = var.vpc_id
name = each.value.count
tags = each.value.tags
}
resource "aws_security_group_rule" "additional_cidr" {
for_each = {
for o in local.cidr_rules : "${o.group_key}:${o.rule_key}" => o
}
security_group_id = aws_security_group.additional[each.value.group_key]
type = each.value.rule.type
protocol = each.value.rule.protocol
from_port = each.value.from_port
to_port = each.value.to_port
cidr_blocks = each.value.cidr_blocks
description = each.value.description
}
resource "aws_security_group_rule" "additional_source_group" {
for_each = {
for o in local.source_group_rules : "${o.group_key}:${o.rule_key}" => o
}
security_group_id = aws_security_group.additional[each.value.group_key]
source_security_group_id = aws_security_group.additional[each.value.rule.source_group_key]
type = each.value.rule.type
protocol = each.value.rule.protocol
from_port = each.value.from_port
to_port = each.value.to_port
description = each.value.description
}
So with all of this together what we’ve achieved is a module which can take a hierarchical description of various security groups and their rules, where some rules possibly refer to other security groups, and produce the flat set of security group and rule declarations to produce that result with all of the needed interconnections.
This module doesn’t manage a VPC and it doesn’t do anything with CSV files. It could do those things, but I suggest keeping each module focused on a particular goal and then connecting multiple modules together in order to construct a larger system. In this case, perhaps you’d have a separate module that declares a VPC, but for the sake of keeping things simple to wrap this up I’m going to just assume that the root module will directly declare a VPC and pass it in to the vpc_id variable I declared above.
One way to use this module now would be to eschew the CSV files altogether and to just write the definitions out directly inside the module block:
resource "aws_vpc" "example" {
# ...
}
module "security" {
source = "./modules/vpc-security"
vpc_id = aws_vpc.example.id
default_security_group = {
manage_ingress = true
manage_egress = true
}
security_groups = {
"Synoptek-Edge" = {
name = "Synoptek Edge"
tags = { Name = "Synoptek Edge" }
cidr_rules = {
"internal" = {
type = "ingress"
protocol = -1
from_port = 0
to_port = 0
cidr_blocks = ["10.100.0.0/16"]
description = ""
},
"AWS" = {
type = "ingress"
protocol = "tcp"
from_port = 3389
to_port = 3389
cidr_blocks = ["10.100.10.81/32"]
description = "AWS"
},
# (and so on)
}
source_group_rules = {}
}
}
}
However, you can still use your CSV strategy with this module, by having the calling module dynamically construct the security_groups data structure based on the CSV files it finds on disk:
locals {
rule_files = fileset("${path.module}/security-groups", "*.csv")
rule_file_names = {
# Trim the .csv suffix
for fn in local.rule_files : fn => substr(fn, 0, length(fn) - 4)
}
rule_file_contents = {
for fn in local.rule_files : fn => csvdecode(file("${path.module}/security-groups/${fn}"))
}
rule_file_cidr = {
for fn, raws in local.rule_file_contents : fn => [
for raw in raws : raw
if try(raw.cidr_blocks, "") != ""
]
}
rule_file_srcgrp = {
for fn, raws in local.rule_file_contents : fn => [
for raw in raws : raw
if try(raw.source_group, "") != ""
]
}
groups = {
for fn, raw in local.rule_file_contents : local.rule_file_names[fn] => {
name = local.rule_file_names[fn]
tags = { Name = local.rule_file_names[fn] }
cidr_rules = {
for raw in local.rule_file_cidr[fn] : raw.name => {
type = raw.type
protocol = raw.protocol
from_port = tonumber(raw.from)
to_port = tonumber(raw.to)
cidr_blocks = split(" ", raw.cidr_blocks)
description = try(raw.description, raw.name)
}
}
source_group_rules = {
for raw in local.rule_file_srcgrp[fn] : raw.name => {
type = raw.type
protocol = raw.protocol
from_port = tonumber(raw.from)
to_port = tonumber(raw.to)
source_group_key = raw.source_group
description = try(raw.description, raw.name)
}
}
}
}
}
The above will search a subdirectory of the module directory for files named with a .csv suffix and create a security group object for each one, with a nested rule for each row in the CSV file that has either cidr_blocks or source_group set. I made a few assumptions here that I want to be explicit about:
- This assumes that all of your CSV files would have an additional column
name which is a unique key for each rule within a particular file. It’ll use that as the key in one of the two rule maps.
- I assumed that a particular rule is either CIDR-based or source-group-based, never both at the same time. If it’s source-group-based then I assumed you’d indicate that by writing the unique name of the other group in that field, which would be the same as the filename defining the other group but without the
.csv suffix.
I’ve also just written all of this code directly into the comment box and not tested it against real Terraform, so I expect I’ve made some syntax errors and other mistakes along the way. Hopefully any error messages Terraform returns will give you a hint about how to fix my errors. If not, feel free to post another comment with the full text of the error and I’ll try to explain what mistake it’s reporting.
I hope this helps show some different patterns that’ll be useful when building your module! Of course, you don’t have to take exactly what I showed here for your final module, but I tried to show a few different common techniques here which come up in many reusable Terraform module designs, so you can take those patterns to use in future modules too.
