I created a bunch of workspaces like app-vpc-dev, app-rds-dev, app-vpc-stage, app-rds-stage, etc. on Terraform Cloud using the tfe provider. In order to move this state to Terraform Cloud as well, I created app-terraform-cloud-dev, app-terraform-cloud-stage, etc. using the tfe provider. Ignoring the recursive issue of storing this state too in Terraform Cloud, I migrated the state of workspaces app-vpc-dev, etc. to the workspaces app-terraform-cloud-dev, etc. respectively. However, when I then add a tfe_varaible to any workspace and apply the app-terraform-cloud-dev workspace, I get a bunch of “resource not found” errors. I assume the resources in question are the workspaces to which variables are being added, but the error message is not clear. Does anyone know the issue here? Please ask if anything is not clear or if you would like sample code. If what I’m trying to do is not possible for some reason I would also greatly appreciate any advice.
Hello, and welcome to the forum!
I created app-terraform-cloud-dev, app-terraform-cloud-stage, etc. using the tfe provider. Ignoring the recursive issue of storing this state too in Terraform Cloud…
Creating a ‘bootstrap’ or ‘meta’ workspace from which to provision other TFC workspaces is a common pattern! And while it’s pretty meta, it’s not (or shouldn’t be, anyway) recursive or you’ll run in to the sort of problems that I think you may be experiencing here.
I migrated the state of workspaces app-vpc-dev, etc. to the workspaces app-terraform-cloud-dev, etc. respectively
I’m not sure I understand this correctly - I read that as you having combined the configuration/states of your ‘meta’ workspace and application ones?
In a nutshell, you may be trying to provision a variable to a remote workspace (using tfe_variable) and use it at the same time, which doesn’t work and is moot - just use a variable value itself (either within the configuration or via *.auto.tfvars if you’re using the CLI. Otherwise, leave the workspaces separate and provision the environment of the other first, triggering the application ones second if necessary.
Hope that helps!
Hi Chris,
Thanks for your response!
I have a repo app-terraform-cloud which contains the tfe code separate from my app-terraform repo which contains my AWS Terraform code. In app-terraform-cloud I create a bunch of tfe_workspace such as app-vpc-dev, app-rds-dev, app-vpc-stage, etc. also using Terraform workspaces for each environment. I then have a directory app-terraform-cloud/self where I create the Terraform Cloud workspaces app-terraform-cloud-dev, app-terraform-cloud-stage, etc. I then migrated the Terraform state of the tfe_workspaces in the root of app-terraform-cloud to these workspaces. The tfstate of app-terraform-cloud/self is local, which is what I meant by not migrating that too. When I add a tfe_variable to one of my workspaces such app-terraform-vpc-dev and apply it via the app-terraform-cloud-dev workspace in the UI, that is when I get the “resource not found” errors. If it is a relevant detail, I actually use a module to create the Terraform Cloud workspaces app-vpc-dev, app-rds-dev, etc. which contains a tfe_workspace and AWS credentials as tfe_variables.
To summarize my objective: I first created Terraform Cloud workspaces via a separate repo. This saved me a lot of labor and also allowed me to track my Terraform Cloud workspaces. However, I was left with the local tfstate of many Terraform Cloud resources, including tfe_varaiables which contain sensitive information. I wanted to also use Terraform Cloud to store the state of my tfe resources, so I created additional Terraform Cloud workspaces to hold this state. I also thought I might be able to continue to modify my tfe resources through the Terraform Cloud UI and these all-encompassing Terraform Cloud workspaces.
terraform-tfe-workspace/main.tf
resource "tfe_workspace" "this" {
name = "${var.name}-${var.env}"
...
}
resource "tfe_variable" "aws_access_key_id" {
key = "AWS_ACCESS_KEY_ID"
value = var.aws_access_key_id
category = "env"
workspace_id = tfe_workspace.this.id
sensitive = true
}
...
app-terraform-cloud/main.tf
module "vpc_workspace" {
source = "app.terraform.io/organization/workspace/tfe"
name = "app-vpc"
aws_access_key_id = var.aws_access_key_id
env = var.env
...
}
...
app-terraform-cloud/self/main.tf
name = "app-terraform-cloud-${terraform.workspace}"
...
}
resource "tfe_variable" "aws_access_key_id" {
key = "aws_access_key_id"
value = var.aws_access_key_id
category = "terraform"
sensitive = true
workspace_id = tfe_workspace.this.id
}
resource "tfe_variable" "env" {
key = "env"
value = terraform.workspace
category = "terraform"
workspace_id = tfe_workspace_this.id
}
...
Today I solved the problem. My tfe Terraform code was missing the provider tfe {} block and I also needed to set the TFE_TOKEN environment variable so my bootstrap workspaces could look up my other workspaces.
The only thing I am unsatisfied with is that my bootstrap workspaces (which I put under “self”) have their state local, and I can’t think of a way to get them to Terrraform Cloud without the recursive issue I mentioned.