Is it a good way to use Multiple Backend S3?

I use multiple backend s3 like under

$ terraform init -backend-config=backend-configs/dev.hcl -migrate-state
$ terraform workspace select dev
$ terraform apply

$ terraform init -backend-config=backend-configs/prod.hcl -migrate-state
$ terraform workspace select prod
$ terraform apply

but this way has a problem.
using -migrate-state option, tfstate is created at dev S3, prod S3.

example) when apply dev, prod.tfstate is created at dev S3, but I don’t want it

this is dev S3 path. env:/prod is created by -migrate-state
스크린샷 2024-02-08 오전 10.19.57

this way is good?
or
there is other way better?

https://dev.to/klenam_/working-with-workspaces-and-backends-in-terraform-2ja2

It’s not easy to tell here as you don’t show the content of your backend config file. But it seems you are using a completely separate backend (s3 bucket) for dev and prod. Presumably to keep them isolated and with different access controls?

If this is the case then I don’t think there is any need to use the -migrate-state option. The -migrate-state option will attempt to copy existing state to a new backend.

When you are running the init the config is directing terraform where to look for the shared state. When this occurs then the workspace is set to the ‘default’ workspace. Technically there is no need (if you are using completely different buckets) to then change the workspace as the prod and dev are already isolated. But switching to a named workspace causes no issues.

At this point terraform will work against the dev or prod state backend as per your config.

When you pass terraform a different backend config (eg when you move from working with dev to prod) the migrate state option is causing terraform to ‘copy’ the state from backend configured in the config you were using previously (in this example dev), to the backend configured in the one you are running at that point (prod). The same effect occurs in the opposite direction.

In short - you should not need the -migrate-state option unless you are moving where you are storing the state for a given configuration (which I don’t believe is what you are trying to achieve here)

Happy Terraforming

Thank you :slight_smile:

But, It’s not I wish to know…

Absolutely you are right. I want to do like you said.

this is a flow I want to do.

you told me that don’t need to use the -migrate-state, but terraform cli say “you should use -migrate-state or -reconfigure

can I ignore it?

$ terraform init -backend-config=backend-configs/xxx.hcl
# use `-migrate-state` or `-reconfigure`

ps. I use workspace for using ${terraform.workspace}

provider "aws" {
  profile = terraform.workspace == "dev" ? "dev-role" : "prod-role"
  region  = "ap-northeast-2"

  default_tags {
    tags = {
      terraform   = "true"
      environment = terraform.workspace
    }
  }
}
``

The behaviour you are seeing here is because of a local copy/cache of the backend configuration that is created. If you ever wondered why you only need to provide the -backend-config to the init and not the plan or apply this is why :slight_smile:

As per the documentation:

After you initialize, Terraform creates a .terraform/ directory locally. This directory contains the most recent backend configuration, including any authentication parameters you provided to the Terraform CLI

The fact you are seeing that message leads me to think you are running the INIT on your local computer or on another machine (or pipeline agent) that is not ‘cleaning’ the directory at each run. Therefore Terraform sees that you have an already configured ‘backend’ and prompts you to either:

  1. -migrate - Which will copy the state from the ‘old’ to the new backend
    or
  2. -reconfigure - Which will update the local backend configuration with the new configuration

To get the outcome you desire you can resolve this via

  1. removing the ./.terraform directory entirely, or the ./.terraform/terraform.tfstate file before the first init when changing the backend config
  2. Using the -reconfigure option on your init (probably preferred/simpler)

See the below for an illustration of both methods.

 C:\..\..\..\temp  terraform init -backend-config="a.azurerm.tfbackend"

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
...
Terraform has been successfully initialized!


 C:\..\..\..\temp  ls

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          08/02/2024    17:43                .terraform
-a---          08/02/2024    17:35           2263 .terraform.lock.hcl
la---          08/02/2024    17:33            253 a.azurerm.tfbackend
la---          08/02/2024    17:33            251 b.azurerm.tfbackend
la---          08/02/2024    17:20            444 main.tf

 C:\..\..\..\temp  terraform init -backend-config="b.azurerm.tfbackend"

Initializing the backend...
╷
│ Error: Backend configuration changed
│
│ A change in the backend configuration has been detected, which may require migrating existing state.
│
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".
╵

 C:\..\..\..\temp  rm -r .\.terraform\
 C:\..\..\..\temp  terraform init -backend-config="b.azurerm.tfbackend"

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes....

Terraform has been successfully initialized!


 C:\..\..\..\temp  terraform init -backend-config="a.azurerm.tfbackend"

Initializing the backend...
╷
│ Error: Backend configuration changed
│
│ A change in the backend configuration has been detected, which may require migrating existing state.
│
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".
╵

 C:\..\..\..\temp  terraform init -backend-config="a.azurerm.tfbackend" -reconfigure

Initializing the backend...

Terraform has been successfully initialized!

Note that the -reconfigure can still be present even if you are not changing your backend config on that run.

Hope that helps

Happy Terraforming!

1 Like

Wow… Thank you for the kind explanation! I completely understand now!

Thank you once again :slight_smile:

I have one more question, is it okay to use S3 backends from different AWS accounts? Or is it better to use a single S3 backend and separate them by Workspaces?

First approach) S3 backends from different AWS accounts + Workspaces Dev, Prod
Second approach) Single S3 backend + Workspaces Dev, Prod

Not a problem, happy to have helped.

The answer to your last question is very much ‘it depends’… on both your security and isolation requirements, organisation policies (SDLC, Security, SoD, etc.).

At its most simple, using a single storage resource in a single account which is shared for all terraform projects & deployments, and each deployment uses ‘workspaces’ to isolate the deployments between differing environments. However, this means that you have to give access to that resource to accounts who are developing or deploying. As the state can contain sensitive information (which is accessible in plain text by anyone with access to the state files) this is often not ideal for unauthorised people to be accessing state from production or other projects)

The next is possibly splitting ‘PROD’ and ‘Non-PROD’ environments for all projects between two different storage resources & accounts. This then allows you to set the security differently, ensuring that prod is ‘controlled’ more than non-prod. Using different security principals between the backends. But this still means that a team working on one project may still be able to access state from another project (which may not be desirable)

Then it can get progressively more complex but with increasingly more fine-grained access control and isolation:

Different back-end state storage per project for prod and non-prod
Different back end state storage per environment, per project (dev,test,acceptance, prod…)
etc…

Again, a consideration as to your GitOps and deployment pipelines, not just devs and engineers, related to how they apply to the project and the environments, and the use of differing credentials to ensure isolation will feed into what is best in your circumstances.

1 Like

This topic has been a concern of mine for a long time.

I appreciate you sincerely!