Blue /Green deployments Article

Terraform
1

Hey,

I’ve read the following article on Blue Green deployments:

It appears to add green specific config to the existing configuration which makes use of feature toggles to decide when to create/destroy the green.

After creation of the green infrastructure it appears to delete all the blue config from the .tf file before re applying. This seems really messy?

Am I missing something?

How about keeping the 2 infrastructure definitions and having 2 feature toggles. This way the config doesnt need to change.

Not changing the config for each deployment matters to me since the config I will be using this for multiple instances of these environements,

Any thoughts?

2

Hi @listentorick!

I think when discussing this sort of gradual rollout strategy it’s worth making a distinction between two different situations:

  1. Using a two-stage rollout strategy for a Terraform module itself, where there’s a new version of the module available and we want to create an instance of the new version first, observe it working, and then pivot to it and destroy the old version.
  2. Using a two-stage rollout strategy for separate artifacts like docker images or AMIs, where the structure of the Terraform configuration itself isn’t changing but instead we’re using a multi-step process to pivot from e.g. a compute cluster running some AMI to a compute cluster using a newer revision of that AMI.

For the first of these, there isn’t really any option but to edit the Terraform configuration: you need to specify the new version that you intend to use, and possibly also use its input variables and output values in a different way if its interface has changed. The strategy of temporarily having to module blocks and then removing the older one is, in my personal estimation, generally the best strategy for this situation.

For the second situation, I’d agree that this feels like overkill. When I worked on a system that did something like this for HashiCorp Consul and ElasticSearch VMs, we used a strategy more like what I think you are imagining where there were permanently two instances of the module in the configuration and all that was different between them was an instance count and an AMI id. When we wanted to roll a new version, we’d change the idle module instance to have a nonzero count and the new AMI, apply that to then temporarily have a cluster twice as large with a mixture of versions, let the mechanisms built in to Consul and ElasticSearch sync the existing data over to the new nodes, and then finally set the instance count for the old module instance to zero and apply again to make it idle.

As is often the case with system design, there isn’t a single answer that is right for all cases… it always depends on context. If something like this second solution of pivoting back and forth between two always-defined modules seems like a better fit for your current problem then go for it! If you later have another problem that is shaped more like the first situation I described above, perhaps you’d choose to go the other way for that one.

3

In your discussion of the 2nd scenario, when you say module here do you mean a terraform module or are you referring to the infrastructure as a module? It sounds like the later?

Since renaming infrastructure such as VMs results in destruction (in azure at least) I dont intend to change the VM names from green to blue (for obvious reasons) but rather use tagging. Does this seem appropriate?

4

I meant a module in the Terraform sense of the word. This applies only to a particular subset of modules that contain things that can temporarily coexist during a staged rollout, but it’d look something like this:

module "example_v1" {
  source  = "example.com/example/example"
  version = "1.0.0"

  name_suffix = "v1"
}

module "example_v2" {
  source  = "example.com/example/example"
  version = "2.0.0"

  name_suffix = "v2"
}

The sequence of steps I was describing would have this configuration start off with only module "example_v1", and then during rollout both of them are temporarily present for the first step and then finally module "example_v1" is removed once module "example_v2" is up and running, returning to only having one module.

As you can see, there is no specific mention of “blue” or “green” in this case. This is exploiting the ephemeral nature of cloud infrastructure to bring up totally new infrastructure for each new release, rather than to pivot back and forth between two long-lived instances. In a sense I suppose strictly this is a “blue/green-like” approach, rather than strictly blue/green: the sense of a particular cluster being blue or green is replaced by the idea of “older/newer”, or “previous/next”, with each “generation” of the infrastructure having a version number associated with it.

In the other example of pivoting between two long-lived instances (called “blue” and “green”, probably), in the specific system I was thinking of where I did something like that the module was set up to take a count and an image id as input variables:

variable "blue" {
  type = object({
    image = string
    count = number
  })
  default = {
    image = null
    count = 0
  }
}

variable "green" {
  type = object({
    image = string
    count = number
  })
  default = {
    image = null
    count = 0
  }
}

module "blue" {
  source = "./modules/bluegreen"

  image_id       = var.blue.image
  instance_count = var.blue.count
}

module "green" {
  source = "./modules/bluegreen"

  image_id       = var.green.image
  instance_count = var.green.count
}

In this case, the “steady state” (while a rollout isn’t in progress) is for one of them to have count = 0 set so that nothing exists for it. So in my case I had the image id as part of the objects being created, since I expected them to be recreated each time we rolled stuff out anyway.

If that isn’t true for you and you intend to leave both the blue and green objects running indefinitely then I would think you would use object names containing “blue” and “green” so that the ones that don’t need to be recreated when the image id changes will be undisturbed.