How to apply changes on EC2 insurances one by one

Terraform
1

Hi There,

I have created 5 EC2 insurances with terraform which are part of mongodb cluster.

Now I have requirement to upgrade EC2 insurances type from t3.medium to t3.large and for the same I have made changes in terraform script and trying to apply change. Terraform is applying changes and rebooting all the EC2 insurances at the same time which will create complete downtime for me.
So I want to apply changes one by one with terraform so wanted to know if there is any way to do that.

Please help me with the solution if there is any. Thank you

2

Hi @bharatjarwal,

Would it be acceptable to instead order the changes so that the new instances will be created before destroying the old instances? That would mean that you would temporarily have 10 instances while Terraform is working but should return back to only five instances by the time the apply operation is complete.

You can use the create_before_destroy lifecycle argument to achieve that opposite ordering, if so.

For it to work you’ll need to make sure that the old and new instances don’t conflict in any way. Fortunately EC2 instances don’t have any unique names that might conflict (it’s fine from the API’s perspective to have two instances with the same tags), but you’ll need to make sure that your mongodb cluster can accommodate there temporarily being twice as many instances as normal, and make sure the new ones don’t conflict with the old ones during that period.

I’m not very familiar with MongoDB but I think a potential problem in this case could be that you’d temporarily have the keyspace sharded in a very different way to normal, and if so keys might end up assigned to different shards during the upgrade.

3

If it would not be acceptable to temporarily have a larger cluster then I think the best approach would be to run terraform apply multiple times, each one changing only one of your instances.

For example, if you’re using count to systematically declare five equivalent instances then you could temporarily add a new conditional expression to the configuration based on a variable, like this:

locals {
  mongodb_instance_size_roll = 0
}

resource "aws_instance" "example" {
  count = 5

  instance_type = count.index < local.mongodb_instance_size_roll ? "t3.large" : "t3.medium"
}

If you apply exactly what I showed above then it should result in no change at all, because none of the instances have an index number less than zero. But if you then set mongodb_instance_size_roll to 1 and apply that you should see Terraform plan to replace only the instance with index zero. You can then change it to 2 and apply again, and so on until you reach 5 at which point you’ll have replaced all of the instances. Then you can remove the additional local value and the conditional expression and return to the simpler configuration you started with, except for having "t3.large" instead of "t3.medium".