Dependencies Don't Wait For Provisioners To Complete

Terraform
1

I create an S3 bucket using terraform. After that I need to download a couple of files from JFrog Artifactory to a local folder. One of the files is a zip which is extracted. This local folder will have 3 more folders. I am using local-exec provisioners inside null_resources to download the files. Then I am using aws_s3_objects to upload all these files and folders to the S3 bucket. However, the local-exec provisioner seems to be executing in a separate thread and also, the aws_s3_object resource runs before the local-exec provisioner completes. So, even if I have depends_on the aws_s3_objects for the null_resources, the JFrog files are not getting uploaded to the S3 bucket. This is somewhat related to the issue described here.
https://github.com/hashicorp/terraform/issues/5595
Can anyone please help?

2

Hi @venh.123,

In order to help understand why Terraform is behaving in this way we will need to see the relevant parts of the configuration you are talking about. Can you share that in another post?

It would also be helpful to see the messages Terraform is printing during the apply step, to see what order the different events are happening in exactly.

Thanks!

3

Thanks a lot for the prompt response. However, I am not sure if I can share all the code. But I will try my best. It will be great if you can try with the below sample, somehow and try to reproduce the issue.

resource "aws_s3_object" "files_upload" {
  for_each = fileset(var.files_source_path, "**/*") 
  bucket = var.s3_bucket_name
  key    = each.value
  source = "${var.files_source_path}/${each.value}"
  etag   = filemd5("${var.files_source_path}/${each.value}")

   depends_on = [
    null_resource.custom_tool 
  ]

}

resource "null_resource" "custom_tool" {
  triggers = {
    always_run = "${timestamp()}"
  }
  provisioner "local-exec" {
    command = "jf rt dl \"${var.custom_tool_url}\"  \"${var.download_location}/\" --explode=true --sort-by=created --sort-order=desc --limit=1 --flat=true"
  }
}
4

Hi @venh.123,

Is it true to say that the “custom tool” is modifying content in the var.files_source_path directory?

If so, this design cannot work because the result of functions like fileset are taken at initial configuration evaluation time, rather than at planning time. Any function which interacts with the filesystem is exclusively for interacting with files that are included as a fixed part of your module, and cannot react to changes your configuration makes during the apply step.

Instead you will need to find a different solution to this problem. My initial suggestion would be that Terraform is not designed to be a build tool and is instead focused on managing changes to long-lived infrastructure, and so anything which involves tricking Terraform into “always running” something, rather than converging on a stable state until next time you explicitly change the configuration, is typically better solved in a different way, outside of Terraform.

For example, a typical approach is to have a CI pipeline that builds new artifacts each time source files change, and publishes those artifacts in a specific way that the Terraform configuration is aware of so it can discover the current or latest artifact using a data source. Then the process would be to first run the build pipeline to change the current artifact to a new one and then run Terraform so it can notice that the current artifact has changed, thereby effectively changing the desired state to select the new artifact, and so plan only the reactions to that change of artifact.

There are some other ways to make this work in Terraform, despite it not really being in Terraform’s intended scope. The main thing is to notice that Terraform models side-effects (that is: changes to systems or objects outside of Terraform’s own memory) using resource and data blocks, and the dependency relationships between them. You can in principle exploit this by using a data source to read files from disk, instead of the built-in functions, and therefore Terraform’s normal dependency handling will notice when it’s necessary to wait until the apply step to read the files. I don’t think any of the official Terraform providers have a “list all files matching this glob pattern” data source similar to fileset, but someone in the community may have published one in the registry, or you could build one yourself. I would recommend considering this as a last resort though, and instead trying to separate your build step (outside of Terraform) from your deploy step (using Terraform), so you can use Terraform for the tasks it is designed for and delegate the build step to a more appropriate tool outside.