Is there a way to daisy-chain `depends_on` for resources created using `count` or `for_each`?

Hello and thank you for terraform,

Using terraform, is there a way to daisy-chain depends_on for resources created using count or for_each ? So inject a dependency into the resource graph?

This is for some cluster stacks where the goal is to have only one AWS EC2 instance being modified at a time (no ASG, AWS CF or containers).

I can accomplish this using static variables, see the example below, but I think because depends_on requires …

“A single static variable reference is required: only attribute access and
indexing with constant keys. No calculations, function calls, template
expressions, etc are allowed here.”
https://www.terraform.io/docs/configuration/resources.html#depends_on-explicit-resource-dependencies

I have not been able to daisy-chain depends_on for resources created using count or for_each.

Is there a way to do this using terraform? If not, then are there any plans to add support for doing this?

Or should a terraform generator (such as python-terrascript) be used to automate writing the terraform file?

I do not think terraform will be able to generate terraform DSL and then consume it within the same plan/apply, and I would want to avoid getting terraform to invoke another terraform process.

> cat daisychain.tf
 
provider "aws" {
  profile    = "default"
  region     = "us-east-1"
}

locals {
  ami = ""
#  instance_type = "t2.small"
  instance_type = "t3.small"
}

resource "aws_instance" "node-01" {
  ami                         = local.ami
  instance_type               = local.instance_type
}

resource "aws_instance" "node-02" {
  ami                         = local.ami
  instance_type               = local.instance_type
  depends_on                  = [ aws_instance.node-01 ]
}

resource "aws_instance" "node-03" {
  ami                         = local.ami
  instance_type               = local.instance_type
  depends_on                  = [ aws_instance.node-02 ]
}

So that when the instance_type is changed:

~ instance_type = "t2.small" -> "t3.small"

each EC2 instance is changed sequentially, one instance at a time:

aws_instance.node-01: Modifying... [id=i-...69]
aws_instance.node-01: Still modifying... [id=i-...69, 10s elapsed]
aws_instance.node-01: Still modifying... [id=i-...69, 20s elapsed]
aws_instance.node-01: Still modifying... [id=i-...69, 30s elapsed]
aws_instance.node-01: Still modifying... [id=i-...69, 40s elapsed]
aws_instance.node-01: Modifications complete after 42s [id=i-...69]
aws_instance.node-02: Modifying... [id=i-...45]
aws_instance.node-02: Still modifying... [id=i-...45, 10s elapsed]
aws_instance.node-02: Still modifying... [id=i-...45, 20s elapsed]
aws_instance.node-02: Still modifying... [id=i-...45, 30s elapsed]
aws_instance.node-02: Still modifying... [id=i-...45, 41s elapsed]
aws_instance.node-02: Modifications complete after 43s [id=i-...45]
aws_instance.node-03: Modifying... [id=i-...ff]
aws_instance.node-03: Still modifying... [id=i-...ff, 10s elapsed]
aws_instance.node-03: Still modifying... [id=i-...ff, 20s elapsed]
aws_instance.node-03: Still modifying... [id=i-...ff, 30s elapsed]
aws_instance.node-03: Still modifying... [id=i-...ff, 40s elapsed]
aws_instance.node-03: Modifications complete after 42s [id=i-...ff]
Apply complete! Resources: 0 added, 3 changed, 0 destroyed.

Thanks

Hi @thatsafunnyname,

I think what you are asking here is for a way to do something like the following…

# This is a hypothetical, and does not work in
# Terraform today.

resource "example" "example" {
  count = 5

  depends_on = count.index > 0 ? [example.example[count.index-1]] : []
}

If so, I’m afraid I can only confirm that this isn’t possible. The reason it isn’t possible is that dependencies are between whole resource blocks, not between the individual instances those resource blocks imply. The count and for_each arguments can themselves be derived from other objects, and so Terraform delays evaluating those until it’s already walking through the graph and it’s too late to introduce any additional dependencies. To restate that more generally: any operation involving expression evaluation can happen only after the graph was already constructed. depends_on is not an expression in the normal sense (as you saw), so it can be processed prior to graph construction.

If these sequential dependencies are a hard requirement for your problem then I think the sort of configuration you showed with a series of separate resource blocks is the only way to get that done today. If it’s predictable enough that you can generate it programmatically then all the better.

I’d suggest also considering what behavior you intend for later operations that happen after the initial creation. While the dependency edges you’ve define here can force the ordering you want during the initial create where nothing exists yet, if a later operation calls for replacing only one of those instances then the others won’t be included in the plan at all and so you could end up in a state where, for example, the remote object associated with aws_instance.node-02 is newer than the one associated with aws_instance.node-03. Perhaps that’s unimportant for your use-case but I just wanted to be explicit about it since you didn’t mention what the purpose of this forced ordering is.

If your intent was just that you’d rather not risk having Terraform make changes to multiple objects concurrently, another option you could consider is to run terraform apply with the extra option -parallelism=1, which will force Terraform to serialize the graph processing itself and only visit one node at a time. (More details in the “internals” documentation section Walking The Graph.)

1 Like

Thanks for the quick response, confirmation and explanation.

I had read about the -parallelism flag, but had ruled it out because it has to be set as an option and I did not see support for incorporating it into a module or configuration file. I just noticed in https://github.com/hashicorp/terraform/issues/18974#issuecomment-433082265 and https://www.terraform.io/docs/commands/environment-variables.html#tf_cli_args-and-tf_cli_args_name that it can be set using an environment variable, but I think if I was to depend on this setting I would prefer it to be part of the module or configuration file.

$ export TF_CLI_ARGS_plan="--parallelism=1"
$ terraform plan

A third option you could consider is to represent your system as two resources that you pivot between:

variable "green" {
  type = object({
    ami           = string
    instance_type = string
    count         = number
  })
}

variable "blue" {
  type = object({
    ami           = string
    instance_type = string
    count         = number
  })
}

resource "aws_instance" "green" {
  count = var.green.count

  ami           = var.green.ami
  instance_type = var.green.instance_type
}

resource "aws_instance" "blue" {
  count = var.blue.count

  ami           = var.blue.ami
  instance_type = var.blue.instance_type
}

locals {
  all_instances = concat(
    aws_instance.green,
    aws_instance.blue,
  )
}

This is a Terraform-flavored blue/green deployment setup, where you would briefly have two sets of instances running during a change but would have one of the two clusters with its count set to zero in stable state.

Of course the drawback of this approach is needing to run terraform apply twice during each change: once to create the new cluster, and then once to destroy the old cluster. But on the plus side, when included as part of some surrounding orchestration scripts you can then execute any arbitrary logic you like between those steps to make sure that the new cluster is fully-functional and behaving correctly before destroying the old one, which cannot be guaranteed by any solution that does the destroys and creates all at once in a single operation.

Just to follow up if it helps anyone else. I ended up with something that looks like the below, with null_resource remote-exec provisioners daisy chained between the aws_instances. The null_resources have calls to uuid as triggers, so always run. This produces the desired behaviour where only one of the EC2 instances is modified at a time.

provider "aws" {
  profile    = "default"
  region     = "TBC"
}

locals {
  ami = "ami-TBC"
#  instance_type = "t2.small"
  instance_type = "t3.small"
  subnet_id = "subnet-TBC"
  ssh_user = "TBC"
  ssh_private_key = file("TBC")
  keyname = "TBC"
  ssh_port = 22
  cidr_blocks = ["TBC"]
}

resource "aws_security_group" "daisychain_intra" {
  name   = "daisychain-intra"
  vpc_id = "vpc-TBC"
  ingress {
    from_port = local.ssh_port
    to_port   = local.ssh_port
    protocol  = "tcp"
    cidr_blocks = local.cidr_blocks
  }
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_instance" "node-01" {
  ami                         = local.ami
  instance_type               = local.instance_type
  subnet_id                   = local.subnet_id
  vpc_security_group_ids      = ["${aws_security_group.daisychain_intra.id}"]
  key_name                    = local.keyname
}

resource "null_resource" "node-01-waitfor" {
  depends_on = [ aws_instance.node-01 ]
  triggers = {
    uuid = uuid()
  }
  provisioner "remote-exec" {
    connection {
      type        = "ssh"
      user        = local.ssh_user
      host        = aws_instance.node-01.private_ip
      private_key = local.ssh_private_key
    }
    inline = [
      "echodate() {",
      "    echo 'node-01-waitfor' `hostname` `date +%FT%T` $*",
      "}",
#     INSERT CLUSTER HEALTH CHECK HERE
      "echodate Done waiting"
    ]
  }
}

resource "aws_instance" "node-02" {
  depends_on                  = [ null_resource.node-01-waitfor ]
  ami                         = local.ami
  instance_type               = local.instance_type
  subnet_id                   = local.subnet_id
  vpc_security_group_ids      = ["${aws_security_group.daisychain_intra.id}"]
  key_name                    = local.keyname
}

resource "null_resource" "node-02-waitfor" {
  depends_on = [ aws_instance.node-02 , null_resource.node-01-waitfor ]
  triggers = {
    uuid = uuid()
  }
  provisioner "remote-exec" {
    connection {
      type        = "ssh"
      user        = local.ssh_user
      host        = aws_instance.node-02.private_ip
      private_key = local.ssh_private_key
    }
    inline = [
      "echodate() {",
      "    echo 'node-02-waitfor' `hostname` `date +%FT%T` $*",
      "}",
#     INSERT CLUSTER HEALTH CHECK HERE
      "echodate Done waiting"
    ]
  }
}

resource "aws_instance" "node-03" {
  depends_on                  = [ null_resource.node-02-waitfor ]
  ami                         = local.ami
  instance_type               = local.instance_type
  subnet_id                   = local.subnet_id
  vpc_security_group_ids      = ["${aws_security_group.daisychain_intra.id}"]
  key_name                    = local.keyname
}

resource "null_resource" "node-03-waitfor" {
  depends_on = [ aws_instance.node-03 , null_resource.node-02-waitfor ]
  triggers = {
    uuid = uuid()
  }
  provisioner "remote-exec" {
    connection {
      type        = "ssh"
      user        = local.ssh_user
      host        = aws_instance.node-03.private_ip
      private_key = local.ssh_private_key
    }
    inline = [
      "echodate() {",
      "    echo 'node-03-waitfor' `hostname` `date +%FT%T` $*",
      "}",
#     INSERT CLUSTER HEALTH CHECK HERE
      "echodate Done waiting"
    ]
  }
}

Output from the initial creation:

 > terraform apply
 Plan: 7 to add, 0 to change, 0 to destroy.
 aws_security_group.daisychain_intra: Creating...
 aws_security_group.daisychain_intra: Creation complete after 1s [id=sg-XXXXa0]
 aws_instance.node-01: Creating...
 aws_instance.node-01: Still creating... [10s elapsed]
 aws_instance.node-01: Creation complete after 12s [id=i-XXXX1a]
 null_resource.node-01-waitfor: Creating...
 null_resource.node-01-waitfor: Provisioning with 'remote-exec'...
 null_resource.node-01-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-01-waitfor (remote-exec):   Host: 172.16.0.1
 null_resource.node-01-waitfor (remote-exec):   User: XXXX
 null_resource.node-01-waitfor (remote-exec):   Password: false
 null_resource.node-01-waitfor (remote-exec):   Private key: true
 null_resource.node-01-waitfor (remote-exec):   Certificate: false
 null_resource.node-01-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-01-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-01-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-01-waitfor (remote-exec):   Host: 172.16.0.1
 null_resource.node-01-waitfor (remote-exec):   User: XXXX
 null_resource.node-01-waitfor (remote-exec):   Password: false
 null_resource.node-01-waitfor (remote-exec):   Private key: true
 null_resource.node-01-waitfor (remote-exec):   Certificate: false
 null_resource.node-01-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-01-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-01-waitfor: Still creating... [10s elapsed]
 null_resource.node-01-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-01-waitfor (remote-exec):   Host: 172.16.0.1
 null_resource.node-01-waitfor (remote-exec):   User: XXXX
 null_resource.node-01-waitfor (remote-exec):   Password: false
 null_resource.node-01-waitfor (remote-exec):   Private key: true
 null_resource.node-01-waitfor (remote-exec):   Certificate: false
 null_resource.node-01-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-01-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-01-waitfor (remote-exec): Connected!
 null_resource.node-01-waitfor (remote-exec): node-01-waitfor ip-172-16-0-1 2020-05-13T14:58:59 Done waiting
 null_resource.node-01-waitfor: Creation complete after 11s [id=2117103898532246551]
 aws_instance.node-02: Creating...
 aws_instance.node-02: Still creating... [10s elapsed]
 aws_instance.node-02: Creation complete after 12s [id=i-XXXX44]
 null_resource.node-02-waitfor: Creating...
 null_resource.node-02-waitfor: Provisioning with 'remote-exec'...
 null_resource.node-02-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-02-waitfor (remote-exec):   Host: 172.16.0.2
 null_resource.node-02-waitfor (remote-exec):   User: XXXX
 null_resource.node-02-waitfor (remote-exec):   Password: false
 null_resource.node-02-waitfor (remote-exec):   Private key: true
 null_resource.node-02-waitfor (remote-exec):   Certificate: false
 null_resource.node-02-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-02-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-02-waitfor: Still creating... [10s elapsed]
 null_resource.node-02-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-02-waitfor (remote-exec):   Host: 172.16.0.2
 null_resource.node-02-waitfor (remote-exec):   User: XXXX
 null_resource.node-02-waitfor (remote-exec):   Password: false
 null_resource.node-02-waitfor (remote-exec):   Private key: true
 null_resource.node-02-waitfor (remote-exec):   Certificate: false
 null_resource.node-02-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-02-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-02-waitfor (remote-exec): Connected!
 null_resource.node-02-waitfor (remote-exec): node-02-waitfor ip-172-16-0-2 2020-05-13T14:59:28 Done waiting
 null_resource.node-02-waitfor: Creation complete after 17s [id=7517128880308308254]
 aws_instance.node-03: Creating...
 aws_instance.node-03: Still creating... [10s elapsed]
 aws_instance.node-03: Creation complete after 12s [id=i-XXXX95]
 null_resource.node-03-waitfor: Creating...
 null_resource.node-03-waitfor: Provisioning with 'remote-exec'...
 null_resource.node-03-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-03-waitfor (remote-exec):   Host: 172.16.0.3
 null_resource.node-03-waitfor (remote-exec):   User: XXXX
 null_resource.node-03-waitfor (remote-exec):   Password: false
 null_resource.node-03-waitfor (remote-exec):   Private key: true
 null_resource.node-03-waitfor (remote-exec):   Certificate: false
 null_resource.node-03-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-03-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-03-waitfor: Still creating... [10s elapsed]
 null_resource.node-03-waitfor (remote-exec): Connecting to remote host via SSH...
 null_resource.node-03-waitfor (remote-exec):   Host: 172.16.0.3
 null_resource.node-03-waitfor (remote-exec):   User: XXXX
 null_resource.node-03-waitfor (remote-exec):   Password: false
 null_resource.node-03-waitfor (remote-exec):   Private key: true
 null_resource.node-03-waitfor (remote-exec):   Certificate: false
 null_resource.node-03-waitfor (remote-exec):   SSH Agent: true
 null_resource.node-03-waitfor (remote-exec):   Checking Host Key: false
 null_resource.node-03-waitfor (remote-exec): Connected!
 null_resource.node-03-waitfor (remote-exec): node-03-waitfor ip-172-16-0-3 2020-05-13T14:59:57 Done waiting
 null_resource.node-03-waitfor: Creation complete after 17s [id=3531458885274467497]
 Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Output after changing the instance type from t3.small to t2.small

 > terraform apply
 Plan: 3 to add, 3 to change, 3 to destroy.
null_resource.node-03-waitfor: Destroying... [id=3531458885274467497]
null_resource.node-03-waitfor: Destruction complete after 0s
null_resource.node-02-waitfor: Destroying... [id=7517128880308308254]
null_resource.node-02-waitfor: Destruction complete after 0s
null_resource.node-01-waitfor: Destroying... [id=2117103898532246551]
null_resource.node-01-waitfor: Destruction complete after 0s
aws_instance.node-01: Modifying... [id=i-XXXX1a]
aws_instance.node-01: Still modifying... [id=i-XXXX1a, 10s elapsed]
aws_instance.node-01: Still modifying... [id=i-XXXX1a, 20s elapsed]
aws_instance.node-01: Still modifying... [id=i-XXXX1a, 30s elapsed]
...
aws_instance.node-01: Still modifying... [id=i-XXXX1a, 4m10s elapsed]
aws_instance.node-01: Still modifying... [id=i-XXXX1a, 4m20s elapsed]
aws_instance.node-01: Still modifying... [id=i-XXXX1a, 4m30s elapsed]
aws_instance.node-01: Modifications complete after 4m34s [id=i-XXXX1a]
null_resource.node-01-waitfor: Creating...
null_resource.node-01-waitfor: Provisioning with 'remote-exec'...
null_resource.node-01-waitfor (remote-exec): Connecting to remote host via SSH...
null_resource.node-01-waitfor (remote-exec):   Host: 172.16.0.1
null_resource.node-01-waitfor (remote-exec):   User: XXXX
null_resource.node-01-waitfor (remote-exec):   Password: false
null_resource.node-01-waitfor (remote-exec):   Private key: true
null_resource.node-01-waitfor (remote-exec):   Certificate: false
null_resource.node-01-waitfor (remote-exec):   SSH Agent: true
null_resource.node-01-waitfor (remote-exec):   Checking Host Key: false
null_resource.node-01-waitfor (remote-exec): Connected!
null_resource.node-01-waitfor (remote-exec): node-01-waitfor ip-172-16-0-1 2020-05-13T15:15:08 Done waiting
null_resource.node-01-waitfor: Creation complete after 0s [id=6272035065940286324]
aws_instance.node-02: Modifying... [id=i-XXXX44]
aws_instance.node-02: Still modifying... [id=i-XXXX44, 10s elapsed]
aws_instance.node-02: Still modifying... [id=i-XXXX44, 20s elapsed]
aws_instance.node-02: Still modifying... [id=i-XXXX44, 30s elapsed]
...
aws_instance.node-02: Still modifying... [id=i-XXXX44, 1m0s elapsed]
aws_instance.node-02: Still modifying... [id=i-XXXX44, 1m10s elapsed]
aws_instance.node-02: Still modifying... [id=i-XXXX44, 1m20s elapsed]
aws_instance.node-02: Modifications complete after 1m23s [id=i-XXXX44]
null_resource.node-02-waitfor: Creating...
null_resource.node-02-waitfor: Provisioning with 'remote-exec'...
null_resource.node-02-waitfor (remote-exec): Connecting to remote host via SSH...
null_resource.node-02-waitfor (remote-exec):   Host: 172.16.0.2
null_resource.node-02-waitfor (remote-exec):   User: XXXX
null_resource.node-02-waitfor (remote-exec):   Password: false
null_resource.node-02-waitfor (remote-exec):   Private key: true
null_resource.node-02-waitfor (remote-exec):   Certificate: false
null_resource.node-02-waitfor (remote-exec):   SSH Agent: true
null_resource.node-02-waitfor (remote-exec):   Checking Host Key: false
null_resource.node-02-waitfor: Still creating... [10s elapsed]
null_resource.node-02-waitfor (remote-exec): Connecting to remote host via SSH...
null_resource.node-02-waitfor (remote-exec):   Host: 172.16.0.2
null_resource.node-02-waitfor (remote-exec):   User: XXXX
null_resource.node-02-waitfor (remote-exec):   Password: false
null_resource.node-02-waitfor (remote-exec):   Private key: true
null_resource.node-02-waitfor (remote-exec):   Certificate: false
null_resource.node-02-waitfor (remote-exec):   SSH Agent: true
null_resource.node-02-waitfor (remote-exec):   Checking Host Key: false
null_resource.node-02-waitfor (remote-exec): Connected!
null_resource.node-02-waitfor (remote-exec): node-02-waitfor ip-172-16-0-2 2020-05-13T15:16:46 Done waiting
null_resource.node-02-waitfor: Creation complete after 16s [id=2739072754194011833]
aws_instance.node-03: Modifying... [id=i-XXXX95]
aws_instance.node-03: Still modifying... [id=i-XXXX95, 10s elapsed]
aws_instance.node-03: Still modifying... [id=i-XXXX95, 20s elapsed]
aws_instance.node-03: Still modifying... [id=i-XXXX95, 30s elapsed]
...
aws_instance.node-03: Still modifying... [id=i-XXXX95, 4m20s elapsed]
aws_instance.node-03: Still modifying... [id=i-XXXX95, 4m30s elapsed]
aws_instance.node-03: Still modifying... [id=i-XXXX95, 4m40s elapsed]
aws_instance.node-03: Modifications complete after 4m44s [id=i-XXXX95]
null_resource.node-03-waitfor: Creating...
null_resource.node-03-waitfor: Provisioning with 'remote-exec'...
null_resource.node-03-waitfor (remote-exec): Connecting to remote host via SSH...
null_resource.node-03-waitfor (remote-exec):   Host: 172.16.0.3
null_resource.node-03-waitfor (remote-exec):   User: XXXX
null_resource.node-03-waitfor (remote-exec):   Password: false
null_resource.node-03-waitfor (remote-exec):   Private key: true
null_resource.node-03-waitfor (remote-exec):   Certificate: false
null_resource.node-03-waitfor (remote-exec):   SSH Agent: true
null_resource.node-03-waitfor (remote-exec):   Checking Host Key: false
null_resource.node-03-waitfor (remote-exec): Connecting to remote host via SSH...
null_resource.node-03-waitfor (remote-exec):   Host: 172.16.0.3
null_resource.node-03-waitfor (remote-exec):   User: XXXX
null_resource.node-03-waitfor (remote-exec):   Password: false
null_resource.node-03-waitfor (remote-exec):   Private key: true
null_resource.node-03-waitfor (remote-exec):   Certificate: false
null_resource.node-03-waitfor (remote-exec):   SSH Agent: true
null_resource.node-03-waitfor (remote-exec):   Checking Host Key: false
null_resource.node-03-waitfor (remote-exec): Connecting to remote host via SSH...
null_resource.node-03-waitfor (remote-exec):   Host: 172.16.0.3
null_resource.node-03-waitfor (remote-exec):   User: XXXX
null_resource.node-03-waitfor (remote-exec):   Password: false
null_resource.node-03-waitfor (remote-exec):   Private key: true
null_resource.node-03-waitfor (remote-exec):   Certificate: false
null_resource.node-03-waitfor (remote-exec):   SSH Agent: true
null_resource.node-03-waitfor (remote-exec):   Checking Host Key: false
null_resource.node-03-waitfor (remote-exec): Connected!
null_resource.node-03-waitfor (remote-exec): node-03-waitfor ip-172-16-0-3 2020-05-13T15:21:35 Done waiting
null_resource.node-03-waitfor: Creation complete after 4s [id=3148319144501976088]
Apply complete! Resources: 3 added, 3 changed, 3 destroyed.