Security group rule resource is getting recreated with each TF apply

Hello,

I am adding a new rule to an existing security group by leveraging the following terraform resource.

aws_security_group_rule

Below the code …

data "aws_security_groups" "lambda-base-sg" {
  filter {
    name   = "group-name"
    values = ["cg-lambda-base-sg"]
  }
}

data "aws_security_groups" "rds-base-sg" {
  filter {
    name   = "group-name"
    values = ["cg-aws_rds-base-sg"]
  }
}

resource "aws_security_group_rule" "rds_sg_inbound" {
  type             = "ingress"
  from_port        = var.db_port
  to_port          = var.db_port
  protocol         = "tcp"
  description      = "Inbound rule from lambda function to RDS"
  source_security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
  security_group_id = join("",data.aws_security_groups.rds-base-sg.ids)
 }

The rule is getting destroyed and added back with every TF execution as shown below

  # module.db-secrets-store.aws_security_group_rule.rds_sg_inbound must be replaced
-/+ resource "aws_security_group_rule" "rds_sg_inbound" {
      - cidr_blocks              = [] -> null
      ~ id                       = "sgrule-2344020695" -> (known after apply)
      - ipv6_cidr_blocks         = [] -> null
      - prefix_list_ids          = [] -> null
      ~ security_group_id        = "sg-0cca3e3981cdbbf1c" -> (known after apply) # forces replacement
      ~ source_security_group_id = "sg-072a8290cc1d9b21a" -> (known after apply) # forces replacement
        # (6 unchanged attributes hidden)
    }

Why the rule is getting replaced every time even though I don’t have any changes. Please help.

Regards
rk

Hi @rxxk-cg,

The output shows that the data sources are not able to be read when generating the plan, but we cannot tell from the limited example why that may be. Could you add a more complete example?

One thing that may help, if the aws_security_groups are being created by your configuration, do not use a data source to represent those same groups, pass the values directly from the resources into the module.

Hello jbardin, Thank you for the response. Those Security groups are already exist in the AWS account and owned by our network team. I am trying leverage those existing SG groups just by adding inbound/outbound rules.

Regarding data source not able to read the sg groups, below the output from the
plan

  # module.db-secrets-store.data.aws_security_groups.lambda-base-sg will be read during apply
  # (config refers to values not yet known)
 <= data "aws_security_groups" "lambda-base-sg"  {
      + arns    = (known after apply)
      + id      = (known after apply)
      + ids     = (known after apply)
      + tags    = (known after apply)
      + vpc_ids = (known after apply)

      + filter {
          + name   = "group-name"
          + values = [
              + "cg-lambda-base-sg",
            ]
        }
    }

  # module.db-secrets-store.data.aws_security_groups.rds-base-sg will be read during apply
  # (config refers to values not yet known)
 <= data "aws_security_groups" "rds-base-sg"  {
      + arns    = (known after apply)
      + id      = (known after apply)
      + ids     = (known after apply)
      + tags    = (known after apply)
      + vpc_ids = (known after apply)

      + filter {
          + name   = "group-name"
          + values = [
              + "cg-aws_rds-base-sg",
            ]
        }
    }

Once the resource a built and rules are added to the existing security groups, the second apply forces a replacement. Below output of the second apply.

  # module.db-secrets-store.data.aws_security_groups.lambda-base-sg will be read during apply
  # (config refers to values not yet known)
 <= data "aws_security_groups" "lambda-base-sg"  {
      ~ arns    = [
          - "arn:aws:ec2:us-east-1:206812888581:security-group/sg-072a8290cc1d9b21a",
        ] -> (known after apply)
      ~ id      = "us-east-1" -> (known after apply)
      ~ ids     = [
          - "sg-072a8290cc1d9b21a",
        ] -> (known after apply)
      + tags    = (known after apply)
      ~ vpc_ids = [
          - "vpc-0b75bf2e58a4af923",
        ] -> (known after apply)

        # (1 unchanged block hidden)
    }

  # module.db-secrets-store.data.aws_security_groups.rds-base-sg will be read during apply
  # (config refers to values not yet known)
 <= data "aws_security_groups" "rds-base-sg"  {
      ~ arns    = [
          - "arn:aws:ec2:us-east-1:206812888581:security-group/sg-0cca3e3981cdbbf1c",
        ] -> (known after apply)
      ~ id      = "us-east-1" -> (known after apply)
      ~ ids     = [
          - "sg-0cca3e3981cdbbf1c",
        ] -> (known after apply)
      + tags    = (known after apply)
      ~ vpc_ids = [
          - "vpc-0b75bf2e58a4af923",
        ] -> (known after apply)

        # (1 unchanged block hidden)
    }


  # module.db-secrets-store.aws_security_group_rule.rds_sg_inbound must be replaced
-/+ resource "aws_security_group_rule" "rds_sg_inbound" {
      - cidr_blocks              = [] -> null
      ~ id                       = "sgrule-2344020695" -> (known after apply)
      - ipv6_cidr_blocks         = [] -> null
      - prefix_list_ids          = [] -> null
      ~ security_group_id        = "sg-0cca3e3981cdbbf1c" -> (known after apply) # forces replacement
      ~ source_security_group_id = "sg-072a8290cc1d9b21a" -> (known after apply) # forces replacement
        # (6 unchanged attributes hidden)
    }

I have multiple security groups all of them seems to be getting replaced

Regards
rk

Yes, if the resources are managed outside of this configuration, then a data source is appropriate to get the needed attribute. As for why the data source is unable to be read during the plan, I would still need a more complete example of how the configuration is structured, i.e. what does the module call for "db-secrets-store" look like, and how are it’s input derived.

Just give some additional background

I am standing up an aurora postgresql database and on-boarding some creds into secrets manager via dbp-aws-secrets module
This is how i invoke the secrets manager module from my primary aurora cluster module.

module "db-secrets-store" {
  source = "./modules/dbp-aws-secrets"
  depends_on = [module.rds_cluster_aurora]
  kms_key_alias = var.kms_key_alias   
  mast_name = local.master_username  - Username to on-board into secerets manager
  mast_pasd = random_password.master_password.0.result 
  end_point = module.rds_cluster_aurora.endpoint 
  dbname    = module.rds_cluster_aurora.database_name
  vpc_filter = var.vpc_filter
  tags = module.tags.tags
  force_delete = 1
  db_port = var.db_port
}

Below the module for secrets manager. The purpose of this module is to deploy a custom lambda function on-board the secrets into secrets manager and rotate those passwords on a periodic basis.

I am creating the SG rules so that
Secrets manager can communicate with
Lambda and
Lambda can communicate with
RDS Service

For the sake of clarity i removed some of the code and provided only the relevant one.

locals {
  usr_info = {
   engine = var.engine
   host = var.end_point
   username = var.mast_name
   password = var.mast_pasd
   dbname   = var.dbname
   port = var.db_port
            }
   account_id = data.aws_caller_identity.current.account_id
   }

data "aws_kms_key" "kms_key" {
  key_id = "alias/${var.kms_key_alias}"
  count = var.kms_key_alias == "" ? 0 : 1
}

data "aws_caller_identity" "current" {}


resource "aws_secretsmanager_secret" "rds_master_creds" { - store scerets resource
  
resource "aws_secretsmanager_secret_version" "rds_credentials" { - on-boarded the scerets resource
  
resource "aws_secretsmanager_secret_rotation" "rotate_master_creds" { - rotate the scerets by attaching a lambda function


resource "aws_security_group" "sec_man_endpoint" {
  name            = "cg-secmanager-endpoint"
  description     = "Security Group for Security Manager VPC endpoint"
  vpc_id          = data.aws_vpc.vpc.id
  tags            = var.tags

  lifecycle {
    ignore_changes = [
      tags,
    ]
  }

}

data "aws_security_groups" "lambda-base-sg" {
  filter {
    name   = "group-name"
    values = ["cg-lambda-base-sg"]
  }
}

data "aws_security_groups" "rds-base-sg" {
  filter {
    name   = "group-name"
    values = ["cg-aws_rds-base-sg"]
  }
}

data "aws_vpc" "vpc" {
  filter {
    name   = "tag:Name"
    values = [var.vpc_filter]
  }
}

data "aws_region" "current" {}

resource "aws_security_group_rule" "sec_man_inbound_to_lambda" {
  type             = "ingress"
  from_port        = 443
  to_port          = 443
  protocol         = "tcp"
  description      = "Inbound rule from vpc end point to lambda function"
  source_security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
  security_group_id = aws_security_group.sec_man_endpoint.id  
}

resource "aws_security_group_rule" "sec_man_outbound_to_lambda" {
  type             = "egress"
  from_port        = 443
  to_port          = 443
  protocol         = "tcp"
  description      = "outbound rule from vpc end point to lambda function"
  source_security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
  security_group_id = aws_security_group.sec_man_endpoint.id
 }

resource "aws_security_group_rule" "lambda_sg_inbound" {
  type             = "ingress"
  from_port        = 443
  to_port          = 443
  protocol         = "tcp"
  description      = "Inbound rule from vpc end point to lambda function"
  source_security_group_id = aws_security_group.sec_man_endpoint.id
  security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
 }

resource "aws_security_group_rule" "lambda_sg_outbound" {
  type             = "egress"
  from_port        = 443
  to_port          = 443
  protocol         = "tcp"
  description      = "Outbound rule from lambda sg to Security Manager"
  source_security_group_id = join("",data.aws_security_groups.rds-base-sg.ids)
  security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
 }

resource "aws_security_group_rule" "lambda_sg_outbound_secman" {
  type             = "egress"
  from_port        = var.db_port
  to_port          = var.db_port
  protocol         = "tcp"
  description      = "Outbound rule from lambda point to RDS Database"
  source_security_group_id = aws_security_group.sec_man_endpoint.id
  security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
 }

resource "aws_security_group_rule" "rds_sg_inbound" {
  type             = "ingress"
  from_port        = var.db_port
  to_port          = var.db_port
  protocol         = "tcp"
  description      = "Inbound rule from lambda function to RDS Endpoint"
  source_security_group_id = join("",data.aws_security_groups.lambda-base-sg.ids)
  security_group_id = join("",data.aws_security_groups.rds-base-sg.ids)
 }

resource "aws_vpc_endpoint" "cg_secretsmanager_end" {
  vpc_id            = data.aws_vpc.vpc.id
  service_name      = "com.amazonaws.${data.aws_region.current.name}.secretsmanager"
  vpc_endpoint_type = "Interface"

  security_group_ids = [
    aws_security_group.sec_man_endpoint.id,
  ]
  subnet_ids         = data.aws_subnet_ids.private.ids
  private_dns_enabled = true

  tags              =var.tags

  lifecycle {
    ignore_changes = [
      tags,
    ]
  }

}

data "aws_subnet_ids" "private" {
  vpc_id = data.aws_vpc.vpc.id

  tags = {
    Name = "*priv*"
  }
}

resource "aws_lambda_function" "rds_autora_passwd_rotation" { - to create the lambda function 
  

Thanks for the extra info @rxxk-cg,

Your db-secrets-store module call is using

depends_on = [module.rds_cluster_aurora]

Which means that every object within the db-secrets-store module depends on every object within the rds_cluster_aurora module, hence your data sources depend on any and every change from that other module. There is no reason you are ever required to use depends_on with a module, what was your intent with adding that?

Removing the depends_on line from the module call is likely to fix the problem.

Thank you very much that was the issue. Removing the Depends on clause resolved the issue.