That is, my instance normally has two IP addresses assigned, the main one providing “production” access to a TCP service and the dev one providing a development-oriented version of the services from the same host. (This is TCP, not HTTP so I can’t route based on a Host header, and neither the production nor the development services require much in the way of resources.)
One overarching goal is to keep the deployment simple and cheap as far as day-to-day hosting. But I would like to be able to swap this server out whenever it needs updating with only brief downtime (ideally none but in practice something on the order of 10 seconds should be readily accepted by the rest of the system).
What I’m imagining is basically to prep a new instance (via the dev IP address or some temporary one) and then switch both of the public Elastic IPs over once it’s ready. There may be better or more popular terms but I’ve seen this idea called a blue/green deployment strategy iiuc. Preparation currently involves both a Terraform step to get a “blank” VM followed by an Ansible step to install and configure the software.
How do I accomplish this, routinely, with Terraform? Would it be sensible to try set up my .tf files so that I can step through a changeover something like:
I’ve seen guides involving manual `terraform state mv` commands but it appears that dynamic moved blocks aren’t really a thing yet as just one hurdle I’d have to work around. My impression is that Terraform just isn’t really designed to juggle an outgoing vs. incoming version of one particular resource, is that correct? (n.b. a load balancer instance would add 5x or more on top of what the app instance itself costs to run…)
I really wanted to have this setup as “infrastructure as code” but if HCL really doesn’t support coding this kind of changeover I’m worried I might just need to document and run through a manual process after all :-/
I think the most common way folks set up things like this is to have Terraform manage an autoscaling group instead of managing an EC2 instance directly, but to configure the autoscaling group to have just a fixed number of instances. An advantage of that approach is that you can use the autoscaling system’s built-in “instance refresh” features for rolling between instances matching different launch templates, as in the example Automatically refresh all instances after the group is updated, which I’ll reproduce below just in case those docs get edited in future.
If having just a single aws_instance resource is important though, you can indeed achieve a similar thing using just Terraform features, but because Terraform is a “desired state” system rather than an imperative system I’d suggest framing it a little differently:
This configuration declares aws_instance.app_server["a"] and also declares that Terraform should not try to update instances of this resource after they are created (ignore_changes = all) so that you can safely modify the configuration when you’re intending to perform an instance swap.
When you want to roll to a new instance:
Change instance_keys to be toset(["a", "b"]) and run terraform apply to cause Terraform to create aws_instance.app_server["b"] while retaining aws_instance.app_server["a"] for now.
Once you’ve confirmed that the new instance is working correctly, change active_instance to "b" and run terraform apply to switch the Elastic IP over to the new EC2 instance.
Once you’ve confirmed that the EIP is indeed referring to the new instance, change instance_keys to be toset(["b"]) and run terraform apply to cause Terraform to destroy aws_instance.app_server["a"].
I used "a" and "b" here as placeholders but you can use whatever identifiers you want. For example, you could literally use "blue" and "green" if you want to associate this with blue/green deployment. When I used a strategy like this in the past I instead used a string containing a date and an index, like 20260825.1, just because that helped me to always remember which is the newer instance and see how long it’d been since I last rotated the instances.
I proposed doing this by editing the configuration, but in theory you could do this with input variables on the command line instead if you like. The trick there though is that Terraform has no “memory” of what input variables you used last time, and so you’d need to manually make sure that you preserve the old instance’s key until you’re ready to delete it.
Thanks much! This was a very helpful starting place.
I’ve now got things set up such that I can through a file like state.auto.tfvars manually changing one entry to something like:
deployment_lifecycle_state = "a-to-b"
To enter either a transitional ("a-to-b"/"b-to-a") or what I’m calling the “steady” ("a"/"b") state(s) which are meant to deal with the pesky fact that I need two different sets of private IP addresses that can both be active at the same time.
So basically I have one relatively long-lived server (either the A or B version, doesn’t really matter…) and then when it’s time to update I load up the other version simultaneous—giving it the dev_iface—and then when its ready I swap the main_iface over to it as well when going back to to the new (either B or A…) version of the steady state.
Even seems to be possible to transition from my previous single instance to the new for_each trick without too much trouble:
moved {
from = aws_instance.app_server
to = aws_instance.app_server["my-app-a"]
}
locals {
instance_a = {
name = "my-app-a"
main_iface = var.main_ifaces[0]
dev_iface = var.dev_ifaces[0]
}
instance_b = {
name = "my-app-b"
main_iface = var.main_ifaces[1]
dev_iface = var.dev_ifaces[1]
}
_cycle = var.deployment_lifecycle_state // ("a" / "b" / "a-to-b" / "b-to-a")
main_host = contains(["a", "a-to-b"], local._cycle) ? local.instance_a : local.instance_b
_alt_host = contains(["a", "a-to-b"], local._cycle) ? local.instance_b : local.instance_a
next_host = contains(["a-to-b", "b-to-a"], local._cycle) ? local._alt_host : null
app_instances = local.next_host != null ? {
(local.main_host.name) = local.main_host
(local.next_host.name) = local.next_host
} : {
(local.main_host.name) = local.main_host
}
// which app instance (key) should get the public IP of each subdomain?
active_main = local.main_host.name
active_dev = local.next_host != null ? local.next_host.name : local.main_host.name
}
resource "aws_instance" "app_server" {
for_each = local.app_instances
ami = data.aws_ami.debian.id
key_name = aws_key_pair.deployer.key_name
instance_type = var.instance_type
metadata_options {
http_tokens = "required"
}
…snip…
private_ip = each.value.main_iface
secondary_private_ips = [each.value.dev_iface]
tags = {
Name = each.value.name
}
}
resource "aws_eip" "app_main" {
instance = aws_instance.app_server[local.active_main].id
associate_with_private_ip = local.app_instances[local.active_main].main_iface
}
resource "aws_eip" "app_dev" {
instance = aws_instance.app_server[local.active_dev].id
associate_with_private_ip = local.app_instances[local.active_dev].dev_iface
}
output "address" {
description = "Public IP address of main gateway"
value = aws_eip.app_main.public_ip
}
output "address_dev" {
description = "Public IP address of dev gateway"
value = aws_eip.app_dev.public_ip
}
output "hosts" {
description = "EC2 instances"
value = {for k,v in aws_instance.app_server : k => "${v.id} @ ${v.public_dns}"}
}
output "ip_association" {
description = "Private association of main IP address"
value = aws_eip.app_main.private_ip
}
output "ip_association_dev" {
description = "Private IP association of dev IP address"
value = aws_eip.app_dev.private_ip
}
output "lifecycle_state" {
description = "Current deployment lifecyle state"
value = var.deployment_lifecycle_state
}
Main thing left is to see how it goes on the AWS side e.g. with existing open connections on the outgoing server. As I mentioned originally, the devices downstream of this service should be able to handle a little downtime but I do want to characterize what happens in practice during Elastic IP re-assignment as the AWS docs weren’t 100% clear to me as far as timing/expectations.