How to handle short-lived infrastructure created to support a CI pipeline

Terraform
1

Hi @apparentlymart,

Thank you for the answer.

In my case we create some short lived resources (K8s deployments) as part of a continuous integration (CI) pipeline and then those resources are destroyed after CI is over. While the pipeline is running, the terraform plan done by a developer would show those volatile resources and a developer might even destroy them. Could you maybe suggest some way of resolving that? If not, could you elaborate on your answer to @RoyceTheBiker a bit more? Is it even possible to have a separate Terraform configuration for the same Kubernetes cluster?

2

Hi @mezhaka,

In the situation you have described I expect I would create a separate Terraform configuration which describes only the short-lived resources needed for the pipeline, and use scripting in the CI workflow to ensure that each run of that pipeline has its own distinct Terraform state for that configuration. For example, you could use the -backend-config argument to terraform init to override the state storage location on a per-run basis.

If there are different objects that need to be planned and applied separately then a separate configuration for each group is the typical answer. In this case I’ve also added the extra idea of using a separate state per run so that each run can have its own set of objects that will exist only for the duration of that run in support of that run in particular. The pipeline can end by running terraform destroy to destroy only the objects related to that particular run, which are tracked in its own state snapshot.

Because these are in an isolated configurations with isolated state snapshots for each run, there is no chance of a developer accidentally interfering with this temporary infrastructure. In order to see it in a terraform plan they would need to first check out the separate configuration and intentionally run terraform init with backend config that matches one of the runs already in progress, which could be a useful thing to do if needing to debug or clean up after a failed run, but should not be a routine part of the use of this pipeline.