Create each sg rule for each node group?

Terraform
1

Hey folks, i’ve been banging my head against the wall on this one, and not sure if it is possible.

We are spinning up EKS clusters, and would like to use managed node groups. However, the limitation of managed node groups is that you cannot provide the security group for each node group-- amazon creates one for you.

We are currently barred from using managed node groups because we need to control the ports on the node security groups, but then I thought of a hack I could try that might work (and it sort of does, just not to the extent I would like).

I have a null data source that waits for the node group to be created, then a data block that looks up the security group that amazon created by name, which allows us to provide it in an aws_security_group_rule resource, therefore managing the SG.

Here is the request: I would like to be able to create security group rules for each rule defined within each node group configuration. Here is some example code that I have that is working:

node_groups = {
  default = {
    instance_type = "t3.medium"
    disk_size     = 20
  }
  node_group_2 = {
    instance_type = "m4.large"
  }

additional_nodegroup_sg_rules = {
  k8s = {
    node_group               = "default"
    description              = "app database",
    type                     = "ingress",
    from_port                = 5432
    to_port                  = 5432
    protocol                 = "tcp"
    source_security_group_id = "sg-12345"
  },
  ssh = {
    node_group  = "node_group_2"
    description = "SSH",
    type        = "ingress",
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["10.60.192.0/28"]
  },
}

data "aws_security_group" "node_group_sg" {
  for_each = var.node_groups
  filter {
    name   = "tag:eks"
    values = [aws_eks_node_group.node_group[each.key].node_group_name]
  }
  depends_on = [data.null_data_source.node_groups_sg]
}

resource "aws_security_group_rule" "managed_node_ssh_access" {
  for_each                 = var.additional_nodegroup_sg_rules
  security_group_id        = data.aws_security_group.node_group_sg[lookup(each.value, "node_group", null)].id
  description              = lookup(each.value, "description", null)
  type                     = lookup(each.value, "type", null)
  from_port                = lookup(each.value, "from_port", null)
  to_port                  = lookup(each.value, "to_port", null)
  protocol                 = lookup(each.value, "protocol", null)
  cidr_blocks              = lookup(each.value, "cidr_blocks", null)
  source_security_group_id = lookup(each.value, "source_security_group_id", null)
}

The limitation here is that if the user has a single rule they wish to apply to ALL node groups, it has to be defined multiple times, once for each specific node group. I would like to be able to set the value of node_group within additional_nodegroup_sg_rules to "all" and only have to define it once, but have it propagate down to all the node group sgs.

I have tried every manner I can come up with of manipulating the data to create values and a workflow that supports this, but I just don’t think it is possible.

I think the only way would be to somehow convert the data with a local so that:

additional_nodegroup_sg_rules = {
  ssh = {
    node_group  = "all"
    description = "SSH",
    type        = "ingress",
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["10.60.192.0/28"]
  },
}

turns into a map that looks like:

global_nodegroup_sg_rules = {
  ssh_default = {
    node_group  = "default"
    description = "SSH",
    type        = "ingress",
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["10.60.192.0/28"]
  },
  ssh_node_group_2 = {
    node_group  = "node_group_2"
    description = "SSH",
    type        = "ingress",
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["10.60.192.0/28"]
  },
}

and then have an additional sg rule resource block that runs against local.global_nodegroup_sg_rules

Any ideas?

Thanks in advance!

2

Hi @nomeelnoj!

Hmm… it seems like what you want to do should be doable here, but we’ll need to break down the problem into smaller steps to get it done.

I think what you tried already was on the right track here: whenever I have a collection of items where some subset of them need to be processed in one way and the rest need to be processed another way it’s easier to first pre-process them into two separate collections, so we don’t have to keep re-filtering the input on every subsequent step. With that in mind, let’s create node_group_sg_rules and global_sg_rules, like this:

locals {
  node_group_sg_rules = {
    for k, r in var.additional_nodegroup_sg_rules :
    k => r if r.node_group != "all"
  }
  global_sg_rules = {
    for k, r in var.additional_nodegroup_sg_rules :
    k => r if r.node_group == "all"
  }
}

These two for expressions have exactly opposite if clauses, so all of the elements in the input will end up in one or the other.

As you said, it looks like our next step here would be to now try to expand the map in local.global_sg_rules so that rather than having one entry per rule it has one entry per rule per security group. When we want to find all of the different pairings of elements from two collections, setproduct is a good tool to reach for, but it works with lists and sets rather than maps and so we’ll use the keys from our two maps for the product here and then transform back into a mapping at the end:

locals {
  global_sg_rule_per_sg = {
    for pair in setproduct(
        keys(local.global_sg_rules),
        keys(var.node_groups),
    ) : "${pair[0]}:${pair[1]}" => merge(local.global_sg_rules[pair[0]], {
      # This overrides the node_group = "all" from
      # the original rule definition, individualizing it.
      node_group = pair[1]
    })
  }
}

The expression above is pretty gnarly, so you might want to split it up in to several separate local values in practice, but the key point is to produce a map with a rule for every combination of global security group rule and node group, overriding the node_group attribute (using merge) to make the result compatible with what’s in local.node_group_sg_rules. The keys are compound strings including both source keys, like k8s:node_group_2, where I selected : as the separator because it seemed like none of your input keys were using it already and thus it shouldn’t collide with any non-global keys.

Now we can finally merge those two together to produce the final, expanded set of security group rule objects:

locals {
  expanded_node_group_sg_rules = merge(
    local.node_group_sg_rules,
    local.global_sg_rule_per_sg,
  )
}

The above now meets the expected constraints for a for_each expression: one element per instance we want to create.

resource "aws_security_group_rule" "managed_node_access" {
  for_each = local.expanded_node_group_sg_rules

  security_group_id        = data.aws_security_group.node_group_sg[each.value.node_group].id
  description              = try(each.value.description, null)
  type                     = try(each.value.type, null)
  from_port                = try(each.value.from_port, null)
  to_port                  = try(each.value.to_port, null)
  protocol                 = try(each.value.protocol, null)
  cidr_blocks              = try(each.value.cidr_blocks, null)
  source_security_group_id = try(each.value.source_security_group_id, null)
}

I’m afraid there’s a lot here so I wasn’t able to test all of this before posting it and I’m sure I’ve made some typos or other mistakes somewhere along the line. Hopefully Terraform’s error messages will be helpful to understand what I should’ve written, but if not please let me know the full error messages you see and I’ll try to correct myself.

3

Thank you SO MUCH for this! I wont have a chance to test it for a day or two (in the meantime we are just not supporting global rules in this module) but I will add it as soon as I can and get back to you with the results.

Can’t thank you enough!