How to split up large project into smaller projects

I have a very large project (configuration directory) that I am considering splitting up into several smaller projects. However, I’m not sure how to split up the state file. Re-creating the infrastructure isn’t really an option, because many of the resources would require production downtime to recreate.

Importing isn’t a great solution, because you can only import one resource at a time, which can take a very long time if there are a lot of resources, and not all resource types support import.

And then there is editing the stat file directly, which is error-prone, and the format of the state file isn’t well documented.

Are there any best practices on how to approach this problem?

For such large scale manipulations I find editing the state file directly to be the quickest option (doing state pushes & pulls). However I would only suggest this if you are comfortable with manipulating the JSON internal format - as you say you could easily make an error.

If you aren’t comfortable with this, there are tools that can do it for you (with the caveat that they might have bugs that cause errors themselves) or use the safe option of terraform import.

A technique I have found useful in the past, is to make multiple copies of the existing state file, assign one of them to each new smaller project, and then use terraform state rm to delete all the objects not needed in each smaller project.

This is a lot faster that import, as all of the operations can be done in one batch per split state, and avoids most of the manual editing.

I do, however, usually manually change the lineage field of the state file to a freshly generated UUID in each case, so that Terraform’s built-in safeguard against uploading the wrong state to the wrong place, can still tell the newly split up projects apart.