How to enable message extraction in the firehose cloudwatch decompression feature?

Terraform Providers AWS
1

I want to enable the Extract message fields only from log events → Turn on message extraction flag when configuring the Decompress source records from Amazon CloudWatch Logs feature in a firehose data stream.

I can’t find any reference in the docs.

image
image1230×685 84.8 KB

How to do it?

This issue is halfway what I want. [Docs]: Decompress of log group data is missing · Issue #35443 · hashicorp/terraform-provider-aws · GitHub

2

Looking at ProcessorParameter there doesn’t seem to be a parameter option that might be related to the message extraction setting.

However, I notice on this page that the CreateDeliveryStream API does support a parameter called DataMessageExtraction. The enum in the AWS Go SDK v2 also has DataMessageExtraction, so I think it’s just not documented in the API reference.

Perhaps you can cross-check by creating or updating (if you can, I don’t use Kinesis enough to know) the delivery stream and observe the corresponding CloudTrail event whether DataMessageExtraction is set and what value it’s set to (maybe true, not sure). Then you should be able to set the same parameter in your processors block for decompression in TF.

If you feel charitable, please open an AWS support case and notify them about this missing parameter in their API reference :slight_smile:

Hope this sets you on the right path.

3

Thank you for the help, so even though terraform/aws docs do not specify the use of the CloudWatchLogExport it is already supported in the latest version of the terraform aws provider.

processing_configuration {
      enabled = "true"
      processors {
        type = "Decompression"
        parameters {
          parameter_name  = "CompressionFormat"
          parameter_value = "GZIP"

        }
      }
      processors {
        type = "CloudWatchLogProcessing"
        parameters {
          parameter_name = "DataMessageExtraction"
          parameter_value = "true"
        }
      }
    }

This made the trick. Thank you so much. Gonna open a support case as well :slight_smile:

1 Like
4

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.