Creating aws_mskconnect_connector Getting InvalidParameter issue

Terraform Providers AWS
1

Hello

Terrafoem version = 1.1.3
AWS provier version = 4.9.0

I’m trying to create aws_mskconnect_connector resource.
I’m setting the value of the kafka_cluster.apache_kafka_cluster.bootstrap_servers argument by interpolating the value of aws_msk_cluster.<name>.bootstrap_brokers_tls

resource "aws_msk_cluster" "msk_cluster" {
  cluster_name           = "${var.name}-msk"
  kafka_version          = "2.6.2"
  number_of_broker_nodes = var.number_of_broker_nodes
  enhanced_monitoring    = var.enhanced_monitoring


  broker_node_group_info {
    instance_type = var.instance_type
    client_subnets = var.vpc_private_subnets
    security_groups = [aws_security_group.msk_sg.id]

    storage_info {
      ebs_storage_info {
        volume_size = var.ebs_storage_volume_size
      }
    }

  }

  encryption_info {
    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = true
    }
  }

  open_monitoring {
    prometheus {
      jmx_exporter {
        enabled_in_broker = true
      }
      node_exporter {
        enabled_in_broker = true
      }
    }
  }

  client_authentication  {
    sasl {
      iam = true
      scram = true
    }
  }

  logging_info {
    broker_logs {
      cloudwatch_logs {
        enabled   = true
        log_group = aws_cloudwatch_log_group.cloudwatch_msk.name
      }
    }
  }
}

resource "aws_mskconnect_connector" "db-connector" {
  name = "${var.name}-db-connector"

  kafkaconnect_version = "2.7.1"

  capacity {
    autoscaling {
      mcu_count        = 1
      min_worker_count = 1
      max_worker_count = 2

      scale_in_policy {
        cpu_utilization_percentage = 20
      }

      scale_out_policy {
        cpu_utilization_percentage = 80
      }
    }
  }

  connector_configuration = {
    "connector.class"       = "io.debezium.connector.postgresql.PostgresConnector"
    "tasks.max"             = "1"
    "database.hostname"     = "${var.payments_db_enpoint}"
    "database.port"         = "5432"
    "database.user"         = "${jsondecode(data.aws_secretsmanager_secret_version.db-debezium.secret_string).username}"
    "database.password"     = "${jsondecode(data.aws_secretsmanager_secret_version.db-debezium.secret_string).password}"
    "database.server.name"  = "${var.db-server-name}"
    "name"                  = "${var.name}-db-connector"
    "database.dbname"       = "${var.db-name}"
    "plugin.name"           = "pgoutput"
    "decimal.handling.mode" = "double"
  }

  kafka_cluster {
    apache_kafka_cluster {
      bootstrap_servers = aws_msk_cluster.msk_cluster.bootstrap_brokers_tls

      vpc {
        security_groups = [aws_security_group.msk_sg.id]
        subnets         = var.vpc_private_subnets
      }
    }
  }

  kafka_cluster_client_authentication {
    authentication_type = "IAM"
  }

  kafka_cluster_encryption_in_transit {
    encryption_type = "TLS"
  }

  plugin {
    custom_plugin {
      arn      = aws_mskconnect_custom_plugin.debezium-msk-plugin.arn
      revision = aws_mskconnect_custom_plugin.debezium-msk-plugin.latest_revision
    }
  }

  service_execution_role_arn = aws_iam_role.msk-debezium-secret-access-role.arn
}

I keep getting:

│ Error: error creating MSK Connect Connector (dev-db-connector): InvalidParameter: 1 validation error(s) found.
│ - missing required field, CreateConnectorInput.KafkaCluster.ApacheKafkaCluster.BootstrapServers.

When looking in the state file I see that bootstrap_brokers_tls values were blank, which explains why I’m getting the error.

I’m not sure why it’s empty tough, since encryption_info.0.encryption_in_transit.0.client_broker is set to TLS.

I also see this on the documentation for the aws_msk_cluster resource
AWS may not always return all endpoints so the values may not be stable across applies. Could that be the issue? Am I missing anything?

Thanks :slight_smile: