Hi @darkn3rd,
This is a horrible one 
as the datadog_logs_custom_pipeline resource’s processor block also contains blocks for each processor each with a unique set of attributes (which sometimes are blocks themselves). Although, thankfully, the Nested Schema for processor does define them all.
This would be quite simple if the resource had been written to support Attributes as Blocks for the processor block, but unfortunately it seems not.
My example below takes a sample JSON output (created based upon your example and the docs - as I don’t have the ability to call the datadog API) which is the first 3 processors shown in the datadog_logs_custom_pipeline resource’s example.
It does the following:
- Transforms the JSON into an HCL object which can be used in the
for_each in the dynamic processor block.
- In each iteration of the dynamic block it checks the processor against nested dynamic blocks
- If it matches an included dynamic block it then includes this as dynamic content, setting the attributes
- where a specific processor also includes nested blocks it creates those nested blocks.
I have included a datadog_logs_custom_pipeline.sample_pipeline resource (copied from the doc example) which shows the static (example) vs. the dynamic. From what I can see, the terraform plan is identical.
Copy the below code block into a main.tf in an empty directory, add the relevant datadog provider block and you should be able to run a terraform plan and experiment as needed to explore the solution
locals {
# This is sample JSON output from the pipelines API that contains a list of processors
# that will be used to create the processors in the `datadog_logs_custom_pipeline` resource
# Note It replicates thje first 3 processors from the example at
# https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_custom_pipeline
processors_json = <<JSON
{
"filter": {
"query": "source:python"
},
"id": "string",
"is_enabled": false,
"is_read_only": false,
"name": "",
"processors": [
{
"arithmetic_processor": {
"expression": "(time1 - time2)*1000",
"is_enabled": true,
"is_replace_missing": true,
"name": "sample arithmetic processor",
"target": "my_arithmetic"
}
},
{
"attribute_remapper": {
"is_enabled": true,
"name": "sample attribute processor",
"override_on_conflict": false,
"preserve_source": true,
"source_type": "tag",
"sources": ["db.instance"],
"target": "db",
"target_format": "string",
"target_type": "attribute"
}
},
{
"category_processor": {
"category": [
{ "filter": { "query": "@severity: \".\"" }, "name": "debug" },
{ "filter": { "query": "@severity: \"-\"" }, "name": "verbose" }
],
"is_enabled": true,
"name": "sample category processor",
"target": "foo.severity"
}
}
],
"type": "pipeline"
}
JSON
# Convert the JSON and extract the list of processors and convert to an object of objects (Using merge and the ... expansion syntax)
processors = merge(jsondecode(local.processors_json).processors...)
}
# This resource is taken from the example in the docs (using the first 3 processors)
# This is to show a comparison between the static and dynamic approach
resource "datadog_logs_custom_pipeline" "sample_pipeline" {
filter {
query = "source:foo"
}
name = "sample pipeline"
is_enabled = true
processor {
arithmetic_processor {
expression = "(time1 - time2)*1000"
target = "my_arithmetic"
is_replace_missing = true
name = "sample arithmetic processor"
is_enabled = true
}
}
processor {
attribute_remapper {
sources = ["db.instance"]
source_type = "tag"
target = "db"
target_type = "attribute"
target_format = "string"
preserve_source = true
override_on_conflict = false
name = "sample attribute processor"
is_enabled = true
}
}
processor {
category_processor {
target = "foo.severity"
category {
name = "debug"
filter {
query = "@severity: \".\""
}
}
category {
name = "verbose"
filter {
query = "@severity: \"-\""
}
}
name = "sample category processor"
is_enabled = true
}
}
}
# This resource uses the dynamic block to create the processors
# each processor type is itself a dynamic block with the block
# as each processor type can have different attributes and the
# processor block does not support the attribues as blocks format :(
# https://developer.hashicorp.com/terraform/language/attr-as-blocks
resource "datadog_logs_custom_pipeline" "sample_pipeline_dynamic" {
filter {
query = "source:foo"
}
name = "sample pipeline"
is_enabled = true
dynamic "processor" {
for_each = local.processors
content {
# dynamic block for each processor type (You will need to add more for other processor types)
dynamic "arithmetic_processor" {
for_each = processor.key == "arithmetic_processor" ? [processor.value] : []
content {
expression = arithmetic_processor.value.expression
target = arithmetic_processor.value.target
is_replace_missing = arithmetic_processor.value.is_replace_missing
name = arithmetic_processor.value.name
is_enabled = arithmetic_processor.value.is_enabled
}
}
dynamic "attribute_remapper" {
for_each = processor.key == "attribute_remapper" ? [processor.value] : []
content {
sources = attribute_remapper.value.sources
source_type = attribute_remapper.value.source_type
target = attribute_remapper.value.target
target_type = attribute_remapper.value.target_type
target_format = attribute_remapper.value.target_format
preserve_source = attribute_remapper.value.preserve_source
override_on_conflict = attribute_remapper.value.override_on_conflict
name = attribute_remapper.value.name
is_enabled = attribute_remapper.value.is_enabled
}
}
dynamic "category_processor" {
for_each = processor.key == "category_processor" ? [processor.value] : []
content {
target = category_processor.value.target
# This processor type has a list of categories that are also dynamic!!!!
dynamic category {
for_each = category_processor.value.category
content {
name = category.value.name
filter {
query = category.value.filter.query
}
}
}
name = category_processor.value.name
is_enabled = category_processor.value.is_enabled
}
}
}
}
}
Hope that helps!
Happy Terraforming