How to create AWS Firehose to iceberg destination with multiple tables routing (inline parsing)

Terraform Providers AWS
1

Guys, I am writing a terraform code to create a firehose delivery stream that will write data into iceberg tables. I don’t find anywhere on hashicorp docs/internet an example of routing enabling (inline parsing), represented on aws console UI on section Inline parsing for routing information, enabled via a checkbox, then defining 3 jq expressions. can you please help me on this? many thanks

2

Hello, friend.

Try to use this example above. You need to config the processing_configuration block:

processing_configuration {
  enabled = true
  
  # JQ processor example
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{destinationDatabaseName: .type.table_type, destinationTableName: .type.table_name, operation: \"insert\"}"
    }
  }
1 Like
3

thanks for your reply. i am gonna try this asap and come back here to confirm it works well

1 Like