Hi @christhomas,
It seems like you have a lot of things going on here, so I’m not really sure how to respond.
For the moment, I’m going to focus on something you said at the end of your post, because I believe the rest of it is a consequence of that:
all I did was change terraform resource names, and then suddenly have to recreate the entire VPC
The resource names are how Terraform maintains the correspondence between remote objects (like your VPC in AWS) and the resource blocks in your configuration, so these resource names should generally not be changing routinely.
If you do find the need to change one, you’ll need to also update Terraform’s tracking of the resource in the Terraform state, so that Terraform can understand that you intended to rename your VPC rather than to destroy your existing VPC and create a new one.
The terraform state mv command is that main way to do that. It will directly modify the state, so you’d usually use it right before creating a plan from the configuration that includes the renamed resource:
terraform state mv aws_vpc.old aws_vpc.new
With that said, I’d recommend avoiding renaming resources unless there’s really no other option.
A VPC and its associated subnets are usually such a a fundamental part of any environment that they get created once and never touched again. If you do end up needing to replace them, there isn’t really any straightforward way to describe the sequence of steps required to do that gracefully: you’d want fine control over exactly when different objects transition between the old and new networks to avoid downtime, etc.
The one time I did something like this I did it by temporarily having both the old and new networks existing at once and gradually pivoting individual applications from one network to the other over the course of several weeks. Supporting that required that the “global shared infrastructure” configuration temporarily export two separate sets of network configuration attributes so that all of the “project Bs” (to use your terminology) could intentionally select either one or the other.
Eventually the old network was empty and could be destroyed, at which point the “global shared infrastructure” configuration went back to exporting only one configuration again.
Unfortunately, I don’t really see any way to avoid replacing a VPC being a multi-step complex project. Trying to pivot over everything in a single action would be far too risky for my risk tolerance, regardless of what tooling I was using to do it. Avoiding replacing the VPC altogether is of course the best option, so hopefully the above note about terraform state mv can help keep the same VPC.