We were using aws secret manager to manage credentials of AWS RDS instance. In our code, secret manager is used to save admin username and password of RDS. Code has the block to create secret manager account and create generic secrets for admin username and password. We are using data block to retrieve the created secrets for username and password from the secret manager. Local variable is used to retrieve the username and password as follows
What we observed is that once all the resources have been provisioned, later point of time, if we add/change tag in secret manager resource block, and run terraform apply, it is observed that, version of the username and password stored in the secret manager changes. In effect, RDS instance will be replaced since terraform assume that value of the username is changed because of the version change. But actually, value of the admin user is not changed, but because of the version change, it is assuming that admin username is changed which forces the instance replacement.
So basically, change in tag of secret manager is forcing the instance replacement which is not at all an expected thing to happen. We would like to get expert advice here on this particular scenario.
Thank you for the follow up. Following is output of terraform plan after adding a new tag env = dev to the configuration. There is no other modification is done. Since version of username is getting changed because of tag was added to secret manager, this is forcing db instance to be replaced.
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
~ update in-place
-/+ destroy and then create replacement
<= read (data resources)
Terraform will perform the following actions:
# module.aws_rds.data.aws_secretsmanager_secret.secretmasterDB will be read during apply
# (depends on a resource or a module with changes pending)
<= data "aws_secretsmanager_secret" "secretmasterDB" {
+ arn = (known after apply)
+ description = (known after apply)
+ id = (known after apply)
+ kms_key_id = (known after apply)
+ name = "mysql-secret"
+ policy = (known after apply)
+ tags = (known after apply)
}
# module.aws_rds.data.aws_secretsmanager_secret_version.creds will be read during apply
# (depends on a resource or a module with changes pending)
<= data "aws_secretsmanager_secret_version" "creds" {
+ arn = (known after apply)
+ id = (known after apply)
+ secret_binary = (sensitive value)
+ secret_id = "mysql-secret"
+ secret_string = (sensitive value)
+ version_id = (known after apply)
+ version_stages = (known after apply)
}
# module.aws_rds.aws_cloudwatch_log_group.cloudwatch_log_group["audit"] will be updated in-place
~ resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
id = "/aws/rds/instance/mysql/audit"
name = "/aws/rds/instance/mysql/audit"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (4 unchanged attributes hidden)
}
# module.aws_rds.aws_cloudwatch_log_group.cloudwatch_log_group["error"] will be updated in-place
~ resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
id = "/aws/rds/instance/mysql/error"
name = "/aws/rds/instance/mysql/error"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (4 unchanged attributes hidden)
}
# module.aws_rds.aws_cloudwatch_log_group.cloudwatch_log_group["general"] will be updated in-place
~ resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
id = "/aws/rds/instance/mysql/general"
name = "/aws/rds/instance/mysql/general"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (4 unchanged attributes hidden)
}
# module.aws_rds.aws_cloudwatch_log_group.cloudwatch_log_group["slowquery"] will be updated in-place
~ resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
id = "/aws/rds/instance/mysql/slowquery"
name = "/aws/rds/instance/mysql/slowquery"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (4 unchanged attributes hidden)
}
# module.aws_rds.aws_db_instance.default[0] must be replaced
-/+ resource "aws_db_instance" "default" {
~ address = "mysql.csoqoxbx5arn.ap-south-1.rds.amazonaws.com" -> (known after apply)
~ arn = "arn:aws:rds:ap-south-1:834829128384:db:mysql" -> (known after apply)
~ availability_zone = "ap-south-1c" -> (known after apply)
~ ca_cert_identifier = "rds-ca-2019" -> (known after apply)
~ character_set_name = "" -> (known after apply)
- custom_iam_instance_profile = "" -> null
- customer_owned_ip_enabled = false -> null
- domain = "" -> null
- domain_iam_role_name = "" -> null
~ endpoint = "mysql.csoqoxbx5arn.ap-south-1.rds.amazonaws.com:3306" -> (known after apply)
~ engine_version_actual = "8.0.33" -> (known after apply)
~ hosted_zone_id = "Z2VFMSZA74J7XZ" -> (known after apply)
~ id = "db-PYLIZQTG2VCNW7OHP4PR6564IA" -> (known after apply)
~ identifier_prefix = "" -> (known after apply)
~ latest_restorable_time = "2023-06-28T13:35:00Z" -> (known after apply)
~ listener_endpoint = [] -> (known after apply)
~ master_user_secret = [] -> (known after apply)
+ master_user_secret_kms_key_id = (known after apply)
~ nchar_character_set_name = "" -> (known after apply)
~ network_type = "IPV4" -> (known after apply)
# Warning: this attribute value will no longer be marked as sensitive
# after applying this change.
~ password = (sensitive value)
~ replica_mode = "" -> (known after apply)
~ replicas = [] -> (known after apply)
- replicate_source_db = "" -> null
~ resource_id = "db-PYLIZQTG2VCNW7OHP4PR6564IA" -> (known after apply)
+ snapshot_identifier = (known after apply)
~ status = "available" -> (known after apply)
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ timezone = "" -> (known after apply)
# Warning: this attribute value will no longer be marked as sensitive
# after applying this change.
~ username = (sensitive value) -> (known after apply) # forces replacement
# (37 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.aws_rds.aws_db_option_group.rds_option_group[0] will be updated in-place
~ resource "aws_db_option_group" "rds_option_group" {
id = "mysql-rds-db-option-group"
name = "mysql-rds-db-option-group"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (4 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.aws_rds.aws_db_parameter_group.mysql_db_parameter_group[0] will be updated in-place
~ resource "aws_db_parameter_group" "mysql_db_parameter_group" {
id = "mysql-db-parameter-group-for-instance-mysql"
name = "mysql-db-parameter-group-for-instance-mysql"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (3 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.aws_rds.aws_db_subnet_group.rds_mysql_subnet_group will be updated in-place
~ resource "aws_db_subnet_group" "rds_mysql_subnet_group" {
id = "mysql-db-subnet-group"
name = "mysql-db-subnet-group"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (5 unchanged attributes hidden)
}
# module.aws_rds.aws_iam_role.enhanced-monitoring-IAM-role-rds[0] will be updated in-place
~ resource "aws_iam_role" "enhanced-monitoring-IAM-role-rds" {
id = "mysql-enhanced-monitoring-iam-role"
name = "mysql-enhanced-monitoring-iam-role"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (8 unchanged attributes hidden)
# (1 unchanged block hidden)
}
# module.aws_rds.aws_kms_key.kms_rds_mysql[0] will be updated in-place
~ resource "aws_kms_key" "kms_rds_mysql" {
id = "2a42987f-a826-4f4f-8f1d-06a62e08a7ff"
~ policy = jsonencode(
~ {
~ Statement = [
{
Action = "kms:*"
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::834829128384:root"
}
Resource = "*"
Sid = "Enable IAM User Permissions"
},
- {
- Action = [
- "kms:Encrypt*",
- "kms:Decrypt*",
- "kms:ReEncrypt*",
- "kms:GenerateDataKey*",
- "kms:Describe*",
]
- Effect = "Allow"
- Principal = {
- Service = "logs.ap-south-1.amazonaws.com"
}
- Resource = "*"
- Sid = "Allow CloudWatch Logs to use the key"
},
]
# (2 unchanged elements hidden)
}
)
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (10 unchanged attributes hidden)
}
# module.aws_rds.aws_secretsmanager_secret.secretmasterDB[0] will be updated in-place
~ resource "aws_secretsmanager_secret" "secretmasterDB" {
id = "arn:aws:secretsmanager:ap-south-1:834829128384:secret:mysql-secret-SpV4ud"
name = "mysql-secret"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (5 unchanged attributes hidden)
}
# module.aws_rds.aws_security_group.rds_mysql_security_group[0] will be updated in-place
~ resource "aws_security_group" "rds_mysql_security_group" {
id = "sg-019cceb006847ac5c"
name = "mysql-security-group"
~ tags = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
~ tags_all = {
+ "env" = "dev"
# (2 unchanged elements hidden)
}
# (7 unchanged attributes hidden)
}
Plan: 1 to add, 11 to change, 1 to destroy.
Terraform has explained here (the # forces replacement comment) that this attribute change is what is triggering the replacement.
(The provider is written to replace the instance on all username changes, so I assume this is a necessary property of AWS RDS that it is implementing.)
I’m a bit confused why Terraform thinks the value is being changed at all, why it thinks it will no longer be sensitive, and why it isn’t known at plan time.
Investigating those things would require a lot more visibility of relevant parts of your Terraform configuration.