Unable to set a Json file as Step in EMR

Terraform
1

I’m building an EMR cluster in Terraform and in the STEP argument i want to load a JSON file that describes the list of steps.

I tried this in my main.tf :

ressource "aws_emr" "emr" {
  ...
  ...
  step = "${data.template_file.steps.rendered}"
}

data "template_file" "steps" {
  template = "${file("${path.module}/steps.json")}"
}

And this is the JSON file :

[
{
    "action_on_failure" : "CONTINUE",
    "name"              : "step_name",
    "hadoop_jar_step" : {
        "jar" : "command-runner.jar",
        "args" : [
            "spark-submit",
            "s3://mybucket/src/pyspark/script1.py",
            "1",
            "68465131321321",
            "s3://mybucket/parquet",
            "s3://mybucket/result",
            "321",
            "65165165468587",
            "654"
        ]
    }
}
]

But when i do terraform plan i got this error :

Inappropriate value for attribute “step”: list of object required.

What’s the problem ?

Thanks for your help.

2

step in the aws_emr_cluster resource is considered a block and not an argument. As a result, we aren’t able to pass the JSON object as an argument.

However, with Terraform 0.12, you can use dynamic blocks to read in your steps from JSON and add them as attributes to the nested block:

resource "aws_emr_cluster" "cluster" {
  ...
  dynamic "step" {
    for_each = jsondecode(templatefile("steps.json", {}))
    content {
      action_on_failure = step.value.action_on_failure
      name              = step.value.name
      hadoop_jar_step {
        jar  = step.value.hadoop_jar_step.jar
        args = step.value.hadoop_jar_step.args
      }
    }
  }
  ...
}

Hope this helps!

1 Like
3

Thank you so much for your help. Your solution works like a charm :slight_smile:

Cheers !