[TERRAFORM] - For_each results resource with variable

Terraform
1

Dear,

I am creating some modules for Terraform, but there is a situation in which I am not able to evolve and I would like some help from the community.

In some resource blocks I’m using “for_each” to list and create the resources according to the list.

However, there is one last block in which I need the return of a resource created with for_each and a new for_each with a list as shown below:

resource01 - (main.tf)

resource "aws_glue_job" "glue-job" {
  for_each = var.aws_job_name

  name        = "${var.aws_name}-${each.key}"
  description = local.aws_description
  role_arn    = aws_iam_role.role_glue.arn
  connections = [aws_glue_connection.glue_connection[0].name]
  max_retries = var.aws_max_retries
  timeout     = var.aws_timeout
  default_arguments = {
    "--class"                     = var.aws_arguments_class
    "--additional-python-modules" = var.aws_arguments_python
  }
  command {

    script_location = "s3://${var.aws_s3_bucket}/script/${each.value}"
    python_version  = var.aws_python_version
  }
  execution_property {
    max_concurrent_runs = var.aws_max_runs
  }
  glue_version = var.aws_glue_version
}

resource02 - (main.tf)

resource "aws_glue_trigger" "schedule" {
  for_each = aws_glue_job.glue-job

  name     = "trigger-${each.key}"
  schedule = var.aws_schedule
  type     = "SCHEDULED"

  actions {
    job_name = each.value.name
  }
}

tfvars

  aws_job_name = {
    "process-recommendation"                     = "script01-process-recommendation-v1.py"
    "delete-last-recommendation"                 = "script03-delete-last-recommendation-v1.py"
    "delete-recommendation" = "script10-delete-recommendation-with-more-seven-days-v1.py"
  }
  aws_job_cron = {
    "schedule-process"                    = "0 14,22 * * ? *"
    "delete-last-recommendation" = "0 18,2 * * ? *"
    "schedule-delete-seven-days"          = "0 6 * * ? *"
  }

Scenario:
In resource02 I need to get the “for_each” from the result of resource01 to fill in the fields (name and job_name).

But in resource02 I will also need a “for_each” or something similar to fill the “schedule” field with the value that is in the tfvars variable (aws_job_cron).

However, it is only possible to perform 1 for_each per block, this is where I am, that is, in 1 precise resource:

  • Get the result of a previous block that was created with for_each
  • Use another for_each on top of a variable
2

So am I correct in thinking that you are wanting to end up with three instances of the aws_glue_job resource and nine instances of the aws_glue_trigger resource?

If so, take a look at using setproduct (https://developer.hashicorp.com/terraform/language/functions/setproduct). Create a local map which has the 9 combinations of your two map keys, which you can then use to lookup the correct values in the for_each for the aws_glue_trigger.

4

Hi @stuart-c ,

No,

I want 3 aws_glue_job and 3 aws_glue_trigger

In resource02, in the field (job_name = each.value.name) I will use the result of resource01.

and in the field (var.aws_schedule) within the same resource02 I will also need a for_each for the aws_job_cron variable.

Since I can’t perform 2 for_each on the same block, this is where I’m struggling.

SCENARIO:

1 - resource01 - Creation of 3 aws_glue_job. I’m creating using for_each .

2 - resource02 - Creation of 3 aws_glue_trigger.

I’m reading via for_each the previous block aws_glue_job.glue-job to fetch the creation result. Because I will need to fill in the value in the field (job_name = each.value.name).

In addition to this for_each from the previous block, I also need to perform a new for_each for the field (schedule = var.aws_schedule).

Because each aws_glue_job will have an aws_glue_trigger attached and with a different value for_each.

5

The schedule attribute in the aws_glue_trigger resource accepts a string, so I’m not really sure what you are wanting. Which of the three values from var.aws_job_cron should it be using?

6

Step 1:
This aws_glue_trigger block will create a schedule in each previously created Glue Job.

But for this to happen I have to get the result of the aws_glue_job block and fill it in the field:

   actions {
     job_name = each.value.name

Step 2:
Each schedule of each Job will have a specific time, example schedule with the name Job01 will have the time *"0 14.22 * * ? ", name Job02 will have the time *"0 18.2 * * ? " and the name Job03 will have the *"0 6 * * ? ".

Final: The problem would be in the "aws_glue_trigger" block because;

In the “job_name” parameter: I bring the result of a for_each from another block, at this point it’s ok.

In the “schedule” parameter: Currently it is as a string, but I would need to read a list and start creating it according to the name or by the order of creation, I would use another for_each for the “schedule” line, but it is not possible

resource "aws_glue_trigger" "schedule" {
  for_each = aws_glue_job.glue-job
  for_each = var.aws_schedule

  name     = "trigger-${each.key}"
  schedule = each.key
  type     = "SCHEDULED"

  actions {
    job_name = each.value.name
  }
}

This is what I would like to do, but it is not possible to perform this type of configuration

for_each = aws_glue_job.glue-job
for_each = var.aws_schedule

7

So you are wanting to create 9 instances of the aws_glue_trigger resource then - 3 cron schedules for each of the 3 jobs ( 3 x 3 = 9).

As previously suggested take a look at the setproduct function.

8

No, there are only 3 of each that will be merged.

Example: 3 security group to attach to 3 ec2 instances, I don’t want to multiply, just attach to each other.

3 SG
3 EC2
Final: 3 SG with EC2

The setproduct function does not work because, as you mentioned, it will multiply, that is, it will create all possible combinations.

What I need is:

JOB_GLUE_01: Only the schedule value will be appended to it JOB_SCHEDULE_01

JOB_GLUE_02: Only the schedule value will be appended to it JOB_SCHEDULE_02

JOB_GLUE_03: Only the schedule value will be appended to it JOB_SCHEDULE_03

9

How do you know which job links with which schedule? There doesn’t seem to be anything in common between the two maps?

10

I don’t know if terraform would understand the order in which the variables were written.

11

No, you can’t access the order in which map keys are written.

Could you explain what you are trying to represent in your tfvars?

What are the keys of the aws_job_cron map supposed to represent?

12

If you want things to be ordered you have to use a list. Maps have no ordering.