Multiple Plan+Apply stages

Terraform
1

I’m trying to provision my environment using terraform, it creates everything from scratch, so in case of disaster I can simply run terraform apply and create all AWS and K8s resources I need.

Unfortunately, I keep hitting same wall. Some resources need to be created before others, furthermore some resources need to be created before planning changes in other resources. Terraform’s PLAN everything > APPLY everything model, does not work very well for me. What seems to be missing (or I don’t know how to set it up) is multiple plan+apply > plan+apply runs so I can create basic infrastructure, deploy next tier and then next tier.

For example, I have aws_eks_cluster resource to provision EKS cluster where my application will run and I have kubernetes_deployment resource for my application.
Logically, I can’t plan kubernetes_deployment before aws_eks_cluster created, because terraform can’t connect to EKS cluster to check what exists and what not. It somehow works when nothing exists, because Kubernetes resources don’t exist on first run, then EKS cluster is provisioned and resources are created… but it quickly fails if terraform decides to re-create EKS cluster, because during planning terraform sees Kubernetes resources and thinks “ok, everything is up-to-date”, then during apply terraform kills cluster and does not re-create Kubernetes resources (they were ok during planning).

Another example, is for_each and count resources, terraform can’t plan these without knowing for_each and count values, which makes sense. You have to run terraform apply with -target to create dependant resources first. It makes sense but shows same issue, you need to create some resources before planning others.

Another example, planning and creating rabbitmq_user resources after provisioning kubernetes_deployment for rabbitmq server. I hit same problem again and again.

I tried to use Terraform Cloud, but it simply does not work because there is no way to use -target argument :frowning:

Then, i tried to run terraform with -target, but it’s very complicated, you need to keep track of “first-tier resources” so you can provision them first (like eks cluster, or resources used in for_each or count operations).

I also tried to separate everything into independent terraform folders terraform apply A, terraform apply B but it created even more mess because there is no way to pass parameters between two terraform folders and I have my application tf code spread among multiple folders: create SSL certificate in one folder, create SSL verification DNS records in other.

Modules don’t work because within one run terraform does PLAN everything + APPLY everything.

Maybe there is a way to handle this, and I would really appreciate if somebody could point it out. But it seems to me terraform lacks some sort of staging system which would PLAN stage 1 > APPLY stage 1 > Plan Stage 2 > Apply Stage 2, everything within single terraform run so I can pass variables around and provision modules partially in each stage.

The way I imagine it with multiple stages (ideally running in Terraform Cloud) would be something like:

$ terraform apply
Plan stage 1: VPC, EKS etc
Apply stage 1: VPC, EKS etc
Plan stage 2: k8s workloads
Apply stage 2: k8s workloads

This way if on stage 1 EKS cluster is re-created, terraform will get correct state during stage 2 planning, so stage 2 apply will create missing resources. Maybe it can be achieved with some sort of plan_depends_on parameter.

1 Like
2

This is a rather common problem and the solution is as you say to separate the resources into two or more configurations.

And the state of the first configuration can be used as input to a later configuration using the terraform_remote_state data source

This works with most backends including s3

2 Likes
3

As @bentterp said, multiple configurations is the common answer to this. I would generalize the answer to talk about data sources in general rather than terraform_remote_state in particular: reading remote state is convenient in that it’s reading something you’ve (presumably) already set up to work with Terraform anyway, but some folks prefer one of the following alternative models:

  • Use data sources to query the required objects directly. For example, if you have a shared VPC created by one configuration and consumed by another then you could use a data "aws_vpc" block to look up the expected VPC using a shared tagging scheme or anything else the AWS API can filter VPCs by.
  • Explicitly publish information about shared resources to a configuration store like AWS SSM Parameter Store, and then use the SSM Parameter data source to retrieve it. This strikes a balance between using terraform_remote_state vs. querying objects directly because the publishing is still explicit but the location where it’s published is not specific to a particular Terraform configuration and so it can be easier to refactor things later, and to consume that same data from systems other than Terraform.

Terraform CLI is focused on the task of planning and applying for a single configuration at a time, so orchestrating multiple runs is outside of its scope: it expects that to be handled by some wrapping orchestration. However, if you choose to use Terraform Cloud then it has some features to help with such orchestration. One particularly relevant example for this situation is Run Triggers, where you can arrange for the completion of an apply in one workspace to trigger a plan in another one.

1 Like
4

Thanks, this is helpful. Although it means I have to split my application modules into pieces. Let me elaborate.

I have terraform configuration for environment, it provisions environment infrastructure (VPC, EKS, Cloudwatch, Cloudwatch Agent, Fluentd agent etc) and installs application modules, terraform modules which describe resources required for each application (RDS, Elasticache, Elasticsearch, Kubernetes Workloads etc). These modules normally hosted in application repository, makes it easier to manage. All these modules use variables and outputs to wire infrastructure and applications.

Following your advice I can separate common infrastructure and application modules into multiple configurations. And, definetely remote state data source will help me wire things.

Maybe I can try auto-discovery using data sources. But I’m afraid it might be somewhat fragile. I want to have reliable way of passing configuration between modules within environment, variables work great because they are typed and can be marked as mandatory. It’s harder to forget something. With auto-discovery, I think, things will go wrong too late, during apply. But I have to try to know for sure.

But back the topic… within application modules I have something like this:

resource "aws_acm_certificate" "ingress" {
  domain_name       = var.ingress_host
  validation_method = "DNS"
  tags              = var.tags

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "validation" {
  count   = length(aws_acm_certificate.ingress.domain_validation_options)
  name    = aws_acm_certificate.ingress.domain_validation_options[count.index].resource_record_name
  type    = aws_acm_certificate.ingress.domain_validation_options[count.index].resource_record_type
  zone_id = var.ingress_zone
  records = [aws_acm_certificate.ingress.domain_validation_options[count.index].resource_record_value]
  ttl     = 60
}

I need to create SSL certificate for application to run on the environment. It takes ingress_host and ingress_zone variables to create load balancer and assign SSL certificate to it. This small configuration does not work without -target because you need to create aws_acm_certificate before you can plan aws_route53_record. So, that means I need to separate my application module into two, which violates idea of incapsulating application resources into modules.

Thanks again for remote state data source, it’s will help me to refactor my setup a little bit, but I still think terraform’s plan-everything apply-everything is flawed by design and for_each and count parameter restrictions prove it.

5

Hi @skolodyazhnyy,

To address your "Maybe I can try … " paragraph first: a design convention for data sources is that a data source that requests a single object will fail if that object doesn’t exist, so they can fail during planning as long as the data source configuration itself is complete enough during planning to resolve it then. (That’s dependent on whether or not you’ve use “known after apply” values in the data block.)

Thanks for sharing the specific problem of ACM certificates. This is indeed a situation where the design of the remote system doesn’t gel well with Terraform’s design, because Terraform wants to be able to tell you how many DNS records it is going to create during planning.

This situation seems particularly unfortunate because as far as I can tell the number of elements in domain_validation_options is systematic: it seems to be one element for the primary domain name given in domain_name and then one additional element for each element of subject_alternative_names. In principle then, the provider could generate a partially-known result for domain_validation_options that has the correct number of elements (to use with count) but has the resource_record_value values themselves marked as “known after apply”. It looks like the AWS provider team is considering changes to this resource so it might be fruitful to share this use-case with them to inform that design.

In the meantime though, if I understood the domain validation mechanism correctly it seems like in your case there should always be exactly one element in domain_validation_options, for validating the hostname given in var.ingress_host. If that is true, you could work around the current limitations in this particular case by rewriting this without count:

resource "aws_route53_record" "validation" {
  name    = aws_acm_certificate.ingress.domain_validation_options[0].resource_record_name
  type    = aws_acm_certificate.ingress.domain_validation_options[0].resource_record_type
  zone_id = var.ingress_zone
  records = [aws_acm_certificate.ingress.domain_validation_options[0].resource_record_value]
  ttl     = 60
}

Or, if you want to generalize this to support SANs too you could derive the count value from the inputs to aws_acm_certificate, rather than the outputs:

locals {
  # Flatten all of the certificate hostnames into a single set
  acm_certificate_hosts = toset(concat(
    [aws_acm_certificate.ingress.domain_name], # always one for the primary name
    aws_acm_certificate.subject_alternative_names, # and the SANs, if any
  ))

  # Project to be a mapping rather than a sequence
  # so that the ordering of the elements is not
  # significant.
  acm_cert_validations = {
    for v in aws_acm_certificate.ingress.domain_validation_options : v.domain_name => v
  }
}

resource "aws_route53_record" "validation" {
  for_each = local.acm_certificate_hosts

  name    = local.acm_cert_validations[each.key].resource_record_name
  type    = local.acm_cert_validations[each.key].resource_record_type
  zone_id = var.ingress_zone
  ttl     = 60
  records = [local.acm_cert_validations[each.key].resource_record_value]
}

As long as the subject_alternative_names argument value is known during planning, something like the above should work because we can now predict how many elements will be in local.acm_certificate_hosts even though local.acm_cert_validations will be unknown until the apply step.

It would of course be ideal of the provider were able to do this calculation itself, and maybe it will in the future! But hopefully the above is useful as a way to get a similar result today within the Terraform language itself, albeit with a lot more verbosity.

1 Like
6

Thanks @apparentlymart , idea with “pre-calculating” count value sounds very good. Although it fully relies on logic behind AWS API which may change one day (they ask you to create 2 validation records per domain for example) and things will go wrong. But it sounds good enough and certainly much better than using -target.

7

Indeed, the risk of that changing is probably why the provider doesn’t try to do this itself. The ACM documentation does state that there will be one entry per domain, and the current structure in the underlying API seems to suggest that domain_name in the validation options is intended to be a unique key for the object, but it’s generally true that we can’t predict how AWS APIs will change in the future.