Null_resource + for_each: process incoming files in the filename sort order

Hi,

Input: incoming files in directory “.\incoming”:

00100.sql
00101.sql
00102.sql
00103.sql
00104.sql
00105.sql

Trying to process incoming files with:

terraform {
  required_version = ">= 0.13"
}

locals {
   input_files = sort(tolist(fileset(".\\incoming", "*.sql")))
   input_files_map = {
      for f in local.input_files: f => join("\\", [".\\incoming", f])
   }
}

resource "null_resource" "incoming_sql_files" {
   for_each = local.input_files_map

   triggers = {
      sequence = each.key
   }

   provisioner "local-exec" {
      interpreter = ["PowerShell", "-Command"]
      command = "Write-Host ${each.value}"
   }
}

, even with disabled parallelism, files being processed in ‘random’ order:

PS C:\> terraform apply -parallelism=1 -auto-approve

null_resource.incoming_sql_files["00103.sql"]: Creating...
null_resource.incoming_sql_files["00103.sql"]: Provisioning with 'local-exec'...
null_resource.incoming_sql_files["00103.sql"] (local-exec): Executing: ["PowerShell" "-Command" "Write-Host .\\incoming\\00103.sql"]
null_resource.incoming_sql_files["00103.sql"] (local-exec): .\incoming\00103.sql
null_resource.incoming_sql_files["00103.sql"]: Creation complete after 0s [id=8013475637114309158]
null_resource.incoming_sql_files["00101.sql"]: Creating...
null_resource.incoming_sql_files["00101.sql"]: Provisioning with 'local-exec'...
null_resource.incoming_sql_files["00101.sql"] (local-exec): Executing: ["PowerShell" "-Command" "Write-Host .\\incoming\\00101.sql"]
null_resource.incoming_sql_files["00101.sql"] (local-exec): .\incoming\00101.sql
null_resource.incoming_sql_files["00101.sql"]: Creation complete after 1s [id=8455477174421328302]
null_resource.incoming_sql_files["00105.sql"]: Creating...
null_resource.incoming_sql_files["00105.sql"]: Provisioning with 'local-exec'...
null_resource.incoming_sql_files["00105.sql"] (local-exec): Executing: ["PowerShell" "-Command" "Write-Host .\\incoming\\00105.sql"]
null_resource.incoming_sql_files["00105.sql"] (local-exec): .\incoming\00105.sql
null_resource.incoming_sql_files["00105.sql"]: Creation complete after 0s [id=737936411081243738]
null_resource.incoming_sql_files["00104.sql"]: Creating...
null_resource.incoming_sql_files["00104.sql"]: Provisioning with 'local-exec'...
null_resource.incoming_sql_files["00104.sql"] (local-exec): Executing: ["PowerShell" "-Command" "Write-Host .\\incoming\\00104.sql"]
null_resource.incoming_sql_files["00104.sql"] (local-exec): .\incoming\00104.sql
null_resource.incoming_sql_files["00104.sql"]: Creation complete after 0s [id=3703604562436926496]
null_resource.incoming_sql_files["00100.sql"]: Creating...
null_resource.incoming_sql_files["00100.sql"]: Provisioning with 'local-exec'...
null_resource.incoming_sql_files["00100.sql"] (local-exec): Executing: ["PowerShell" "-Command" "Write-Host .\\incoming\\00100.sql"]
null_resource.incoming_sql_files["00100.sql"] (local-exec): .\incoming\00100.sql
null_resource.incoming_sql_files["00100.sql"]: Creation complete after 1s [id=6010998834953611068]
null_resource.incoming_sql_files["00102.sql"]: Creating...
null_resource.incoming_sql_files["00102.sql"]: Provisioning with 'local-exec'...
null_resource.incoming_sql_files["00102.sql"] (local-exec): Executing: ["PowerShell" "-Command" "Write-Host .\\incoming\\00102.sql"]
null_resource.incoming_sql_files["00102.sql"] (local-exec): .\incoming\00102.sql
null_resource.incoming_sql_files["00102.sql"]: Creation complete after 0s [id=8858832982545845436]

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Is there a way to force the processing in filename sort order?

Thanks!

Hi @igorchernysh,

The model for multiple instances of a resource is that they are independent and thus can be planned and applied in any order relative to one another. There is no way to change that.

In your case it seems like it would be better to treat the series of Write-Host calls as a single operation as far as Terraform is concerned, by combining them together into a single provisioner call. For example:

resource "null_resource" "incoming_sql_files" {
   triggers = {
      files = join("\n", sort(keys(local.input_files_map)))
   }

   provisioner "local-exec" {
      interpreter = ["PowerShell", "-Command"]
      command = <<-EOT
      %{ for key in sort(keys(local.input_files_map)) ~}
      Write-Host ${local.input_files_map[key]}
      %{ endfor }
      EOT
   }
}

The idea of the above is to generate a single PowerShell script with one Write-Host command for each of the filenames, rather than generating a separate PowerShell script for each filename. That way you can control the order by relying on the fact that PowerShell is a procedural programming language which executes one command at a time, in the order given.

For the benefit of anyone else who finds this comment in future I should note that provisioners are a last resort, and I think ideally in this case it would be better if there were a normal Terraform resource type available for doing this operation so that Terraform can track it better, but since I don’t know of any resource type for importing SQL statements right now it does seem like a provisioner call is a pragmatic way to get this done at the time I’m writing this.

2 Likes

This will work. Thank You for the clarification!