Terraform doesn't execute data external in plan phase

We recently upgraded to terraform 1.0.0. We were at 0.13.7 for a long time.
While the upgrade went fine, a recent change was done in one of the ec2 wrapper module (totally unrelated to “data external”)

That external script for us is used to determine the selected ip addresses (based on our own criterias ) to use while launching ec2 instances in for_each blocks. While the upgrade went fine for most of the instance sets (lets say A, B, C), while upgrading certain other instance sets (e.g. D), terraform started complaining about “The “for_each” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply” for A & B.

Reverting the module versions of D or even A,B,C,D doesn’t solve the problem. The response is inconsistent

│ Error: Invalid for_each argument
│   on .terraform/modules/A/main.tf line 47, in resource "aws_instance" "this":
│   47:   for_each = (var.number_of_instances > 0 && local.keys_rotation_enabled) ? module.common.filtered_iprange_output : toset([])
│     ├────────────────
│     │ local.keys_rotation_enabled is true
│     │ module.common.filtered_iprange_output is a list of string, known only after apply
│     │ var.number_of_instances is 4
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply
│ only the resources that the for_each depends on.

module.common.filtered_iprange_output is just a local variable in the common module that is split(",",data.external.cidr_expander.result.filtered_iprange)

However i do query the state file, that the data external is already calculated

$ terraform state show module.A.module.common.data.external.cidr_expander

data "external" "cidr_expander" {
    id      = "-"
    program = [
    query   = {
        "blacklisted_ips" = jsonencode([])
        "cidr_blocks"     = jsonencode(
        "offset"          = "0"
        "range_hop"       = "1"
    result  = {
        "filtered_iprange" = "['', '', '', '', '']"

I am not sure what behaviour has changed in terraform 1.0.0 from 0.13
Terraform should know the output of module.common.filtered_iprange_output even in the plan phase.
There have been lot of requests that have requested not to run data resources in the plan phase or either to simply refresh it.
I am not sure if any change was done around these that is causing side effects for me.

am guessing it could be an aftereffect Data Resource Lifecycle Adjustments · Issue #17034 · hashicorp/terraform · GitHub

Data source’s arguments are fully known during the plan phase as it just a python script that takes some input and statically generates the output. However my real question is how upgrading of a module D cause this problem to start happening to A & B without any changes to them or having any dependency to D

Its odd also as when removing problematic A & B, terraform does say that it wants to read “data” “external” for some modules during the apply phase. It looks unnecessary as the “data” “external” can be calculated in the plan phase itself

  # module.C.module.common.data.external.cidr_expander will be read during apply
  # (config refers to values not yet known)
 <= data "external" "cidr_expander"  {
      + id      = (known after apply)
      + program = [
          + "python",
          + ".terraform/modules/C.common/scripts/myfuction.py",
      + query   = {
          + "blacklisted_ips" = jsonencode([])
          + "cidr_blocks"     = jsonencode([])
          + "offset"          = "0"
          + "range_hop"       = "1"
      + result  = (known after apply)

I later on removed for_each dependency on A and discovered that it too wants to calculate data external during the apply phase

  # module.A.module.common.data.external.cidr_expander will be read during apply
  # (config refers to values not yet known)
 <= data "external" "cidr_expander"  {
      + id      = (known after apply)
      + program = [
          + "python",
          + ".terraform/modules/A.common/scripts/myfunction.py",
      + query   = {
          + "blacklisted_ips" = jsonencode([])
          + "cidr_blocks"     = jsonencode(
                  + "",
          + "offset"          = "1"
          + "range_hop"       = "0"
      + result  = (known after apply)

There is no secret in myfunction.py. It just takes array of CIDR ranges and break down into array of ip addresses in python.

data "external" "cidr_expander" {
  program = ["python", "${path.module}/scripts/myfunction.py"]

  query = {
    cidr_blocks = jsonencode(var.private_ip)
    blacklisted_ips = jsonencode(var.blacklisted_ip)
    range_hop   = var.ip_range_hop
    offset      = var.start_offset

All the arguments of the query are known in the plan phase itself. Wonder what is stoppping terraform to execute it during plan phase itself.

Interestingly, while experimenting, i see a noop too is not applied during plan phase.

  # module.A.module.common.data.external.noop will be read during apply
  # (config refers to values not yet known)
 <= data "external" "noop"  {
      + id      = (known after apply)
      + program = [
          + "python",
          + "-c",
          + "print('{}')",
      + result  = (known after apply)

whose source is

data "external" "noop" {
  program = ["python", "-c", "print('{}')"]

Hi @sksumit1,

I’m not sure exactly what may be happening here without a complete configuration, but the change in behavior when swapping out the data source inputs is definitely curious.

One thing to note is that you should not be using Terraform v1.0.0 which contains many known bugs and may be contributing the issue. The latest patch version from that release is v1.0.11. If I remember correctly however, there were some larger data source dependency handling improvements in v1.1, so you may want to try and work your way up to that version in order to confirm this is a problem with configuration or not.

1 Like

I am guessing this is your bug fix you are talking of.

Let me try the update

Upgrading to 1.1.7 (latest approved for us), didn’t fix the issue.

I have however uploaded a mini project here
Note since zip , tf and py files are not permitted, i converted them to .txt

Notice i didn’t get the problem here. (It even worked on terraform 1.0.0) But in the larger project, where we have multiple modules etc., terraform decides to defer the calculation of the datasource for some of the modules in the main project.

There is no [depends_on] in the datasource.
Interestingly even after reverting all the code changes to the module (minor and unrelated to data external), terraform keeps throwing the listed error.
Is there any type of caching maintained by terraform somewhere.

test.txt (954 Bytes)
launcher.txt (1.2 KB)

As you point out this will always work isolation. What is most important here, and what you are not showing in the examples, is the context in which you use the module. How are var.private_ip and var.blacklisted_ip derived, and what do the modules calls for module.A and module.common look like?

Both the var.private_ip and var.blacklisted_ip passed via the main module A. Inside the main module its pulled from the tfvar file where its hardcoded.
I have updated the code to be closest to our huge codebase for your reference and give the calls details. This still works in the example.

common.txt (879 Bytes)
launcher.txt (743 Bytes)

I ran the terraform graph and i observed that for the complaining module A it is missing a connection

$ grep "module.common.data.external" dev.dot | grep A
		"[root] module.A.module.common.data.external.cidr_expander (expand)" [label = "module.A.module.common.data.external.cidr_expander", shape = "box"]
		"[root] module.A.module.common.data.external.cidr_expander (expand)" -> "[root] module.A.module.common.var.blacklisted_ip (expand)"
		"[root] module.A.module.common.data.external.cidr_expander (expand)" -> "[root] module.A.module.common.var.ip_range_hop (expand)"
		"[root] module.A.module.common.data.external.cidr_expander (expand)" -> "[root] module.A.module.common.var.private_ip (expand)"
		"[root] module.A.module.common.data.external.cidr_expander (expand)" -> "[root] module.A.module.common.var.start_offset (expand)"
		"[root] module.A.module.common.local.filter (expand)" -> "[root] module.A.module.common.data.external.cidr_expander (expand)"
		"[root] module.A.module.common.local.ip_range (expand)" -> "[root] module.A.module.common.data.external.cidr_expander (expand)"
		"[root] provider[\"registry.terraform.io/hashicorp/external\"] (close)" -> "[root] module.A.module.common.data.external.cidr_expander (expand)"

Another sibling module B that is not complaining

$ grep "module.common.data.external" dev.dot | grep B
		"[root] module.B.module.common.data.external.cidr_expander (expand)" [label = "module.B.module.common.data.external.cidr_expander", shape = "box"]
		"[root] module.B.module.common.data.external.cidr_expander (expand)" -> "[root] module.B.module.common.var.blacklisted_ip (expand)"
		"[root] module.B.module.common.data.external.cidr_expander (expand)" -> "[root] module.B.module.common.var.ip_range_hop (expand)"
		"[root] module.B.module.common.data.external.cidr_expander (expand)" -> "[root] module.B.module.common.var.private_ip (expand)"
		"[root] module.B.module.common.data.external.cidr_expander (expand)" -> "[root] module.B.module.common.var.start_offset (expand)"
		"[root] module.B.module.common.data.external.cidr_expander (expand)" -> "[root] provider[\"registry.terraform.io/hashicorp/external\"]".   (this is missing in A)
		"[root] module.B.module.common.local.filter (expand)" -> "[root] module.B.module.common.data.external.cidr_expander (expand)"
		"[root] module.B.module.common.local.ip_range (expand)" -> "[root] module.B.module.common.data.external.cidr_expander (expand)"
		"[root] provider[\"registry.terraform.io/hashicorp/external\"] (close)" -> "[root] module.B.module.common.data.external.cidr_expander (expand)". (This is missing in module C)

I also noticed that the module C (one that was updated and started causing problem to A is also missing the another similar branch between the nodes

$ grep "module.common.data.external" dev.dot | grep C
		"[root] module.C.module.common.data.external.cidr_expander (expand)" [label = "module.C.module.common.data.external.cidr_expander", shape = "box"]
		"[root] module.C.module.common.data.external.cidr_expander (expand)" -> "[root] module.C.module.common.var.blacklisted_ip (expand)"
		"[root] module.C.module.common.data.external.cidr_expander (expand)" -> "[root] module.C.module.common.var.ip_range_hop (expand)"
		"[root] module.C.module.common.data.external.cidr_expander (expand)" -> "[root] module.C.module.common.var.private_ip (expand)"
		"[root] module.C.module.common.data.external.cidr_expander (expand)" -> "[root] module.C.module.common.var.start_offset (expand)"
		"[root] module.C.module.common.data.external.cidr_expander (expand)" -> "[root] provider[\"registry.terraform.io/hashicorp/external\"]"
		"[root] module.C.module.common.local.filter (expand)" -> "[root] module.C.module.common.data.external.cidr_expander (expand)"
		"[root] module.C.module.common.local.ip_range (expand)" -> "[root] module.C.module.common.data.external.cidr_expander (expand)"

Does some kind of corruption introduced by C is causing A to not create a proper dependency graph?

Taking a subset of edges from a graph like that unfortunately does not give us enough context to make any conclusions. The dependency can just as easily be provided transitively through many other nodes.

The example working implies that there is still some detail missing when compared with the complete configuration, so without that detail we can only begin to guess. All I can offer at this point is advice to look closely at all dependencies of the data source taking into account the rules defined for the data resource lifecycle which can defer the read until apply:

  • If the data source configuration contains any unknown value, it cannot be read
  • A data resource with depends_on cannot be read if any of its dependencies have changes

The behavior of depends_on can itself be influenced by 2 other factors:

  • Using depends_on in a module adds that dependency to every resource contained within that module recursively, which can effectively add depends_on to all data sources below it.
  • A data resource directly referencing a managed resource will act as if it has depends_on referencing that managed resource (this can be worked around by indirectly referencing the managed resource through a local value)

Hopefully that helps clarify what you are looking for. If all these cases are provably not a factor, then we would need a way to reproduce the problem with the latest release of terraform to investigate the issue further.

Thank you for the tip. I have fixed the issue
Key issue was what you described.

  • Using depends_on in a module adds that dependency to every resource contained within that module recursively, which can effectively add depends_on to all data sources below it.

module A had a dependency on module C. It was unnecessary and after removing this, the code works fine.
I still don’t understand why it was causing the issue as

  1. The external datasource of A had no dependency on C
  2. instances of module C were already launched and required no further terraform changes in the plan, so even say if the datasource had the dependency, that dependency should be resolved in the plan phase itself.

Thanks for the help here. I appreciate it.