Managing resource interdependencies between separate projects

I’ve got two projects which are totally isolated from each other and use separate state files and deployment pipelines.

Project A: This is a global shared infrastructure project. It defines the VPC, networks, EIP’s, etc. Everything that isn’t related to an actual project.

Project B: This is an actual project. It consumes the VPC id, subnets, etc. Attaches a load balancer, configures security groups, ECS containers api-gateway for this app, etc, etc.

The problem is that a trivial change in the VPC project A has decided that it wants to recreate the subnets. But this is a problem because there are actually 20+ projects deployed in this VPC and they’re all using the subnets for the load balancers they deploy. So there is a deadlock because of that.

There must be a way to organise this project, but I cannot figure out how to get it to work and also, having to completely pull down 20+ projects and their load balancers, just to reconfigure the VPC a bit with not even a configuration change, all I did was change terraform resource names, and then suddenly have to recreate the entire VPC. It’s too much.

Anybody have any ideas?

Hi @christhomas,

It seems like you have a lot of things going on here, so I’m not really sure how to respond.

For the moment, I’m going to focus on something you said at the end of your post, because I believe the rest of it is a consequence of that:

all I did was change terraform resource names, and then suddenly have to recreate the entire VPC

The resource names are how Terraform maintains the correspondence between remote objects (like your VPC in AWS) and the resource blocks in your configuration, so these resource names should generally not be changing routinely.

If you do find the need to change one, you’ll need to also update Terraform’s tracking of the resource in the Terraform state, so that Terraform can understand that you intended to rename your VPC rather than to destroy your existing VPC and create a new one.

The terraform state mv command is that main way to do that. It will directly modify the state, so you’d usually use it right before creating a plan from the configuration that includes the renamed resource:

terraform state mv aws_vpc.old

With that said, I’d recommend avoiding renaming resources unless there’s really no other option.

A VPC and its associated subnets are usually such a a fundamental part of any environment that they get created once and never touched again. If you do end up needing to replace them, there isn’t really any straightforward way to describe the sequence of steps required to do that gracefully: you’d want fine control over exactly when different objects transition between the old and new networks to avoid downtime, etc.

The one time I did something like this I did it by temporarily having both the old and new networks existing at once and gradually pivoting individual applications from one network to the other over the course of several weeks. Supporting that required that the “global shared infrastructure” configuration temporarily export two separate sets of network configuration attributes so that all of the “project Bs” (to use your terminology) could intentionally select either one or the other.

Eventually the old network was empty and could be destroyed, at which point the “global shared infrastructure” configuration went back to exporting only one configuration again.

Unfortunately, I don’t really see any way to avoid replacing a VPC being a multi-step complex project. Trying to pivot over everything in a single action would be far too risky for my risk tolerance, regardless of what tooling I was using to do it. Avoiding replacing the VPC altogether is of course the best option, so hopefully the above note about terraform state mv can help keep the same VPC.

Hi @apparentlymart,

Thanks for your answer, I’m going to try the mv command, because now I’m stuck in the middle and terraform wants to recreate them and I need to tell terraform to stop, go back, and I’ll try another way instead.

I think the way you suggested is probably the best, there are a lot of things going on, but the core of the issue is this

Project A’s subnets changed
Project B’s have load balancers using those subnets

Both projects are independent, so Project B’s are deadlocking Project A from changing anything.

Seems like your way to create new resources and leave the old ones and migrate across step by step until the old networks are empty is the way to go. I unfortunately realised that this change would involve a lot more work until it was too late and now I’m a bit stuck.

Hopefully the ‘terraform mv’ command can help me out. Thanks!