Terraform when using data sources no stored state was found for the given workspace in the given backend

Terraform
1

I am using terraform to setup an aws resource that relies on output from an earlier resource which has its own state. I am making use of data source in terraform to try and achieve this Data Sources - Configuration Language | Terraform | HashiCorp Developer

State

I have two seperate terraform init/states. One networking and one eks. The state is stored on s3 and i make use of workspaces.

s3://myorg-terraform-state/env:/prod/networking/terraform.tfstate
s3://myorg-terraform-state/env:/prod/eks/terraform.tfstate

networking/output.tf

I have a networking terraform that outputs the following.

output "vpc_id" {
  description = "The ID of the VPC"
  value       = module.vpc.vpc_id
}

output "private_subnets" {
  description = "List of IDs of private subnets"
  value       = module.vpc.private_subnets
}

output "worker_group_mgmt_one_id" {
  description = "The id of worker_group_mgmt_one"
  value       = aws_security_group.worker_group_mgmt_one.id
}

output "worker_group_mgmt_two_id" {
  description = "The id of worker_group_mgmt_two"
  value       = aws_security_group.worker_group_mgmt_two.id
}

EKS requires the above variables so i create a data source to be able to access them.

eks/data.tf

data "terraform_remote_state" "networking" {
    backend = "s3"
    config = {
        bucket = "myorg-terraform-state"
        key    = "networking/terraform.tfstate"
        region =  "eu-west-2"
    }
}

eks/main.tf

I then use this datasource in eks/main.tf

...
data.terraform_remote_state.networking.outputs.private_subnets
...

Problem

However when i run the following command i get an error despite the fact that the state exists on s3 and after checking it i can see the outputs.

terraform workspace select prod
terraform plan -var-file=prod.tfvars

Error: Unable to find remote state

on data.tf line 1, in data “terraform_remote_state” “networking”:
1: data “terraform_remote_state” “networking” {

No stored state was found for the given workspace in the given
backend.

Folder Structure

├── networking
│   ├── backend.tf
│   ├── main.tf
│   ├── output.tf
│   ├── prod.tfvars
│   ├── provider.tf
│   └── variables.tf
├── eks
│   ├── backend.tf
│   ├── data.tf
│   ├── main.tf
│   ├── output.tf
│   ├── prod.tfvars
│   ├── provider.tf
│   └── variables.tf
├── README.md
2

Hi @kaykhancheckpoint,

If I’m understanding your S3 paths correctly, it seems like your state snapshots in S3 belong to a workspace called “prod” rather than to the “default” workspace. Since your terraform_remote_state configuration doesn’t mention a workspace, I expect it’s defaulting to the workspace default and thus not finding the state snapshot you were intending.

You can set the workspace argument to specify which workspace of the other configuration you want to read from:

data "terraform_remote_state" "networking" {
    backend   = "s3"
    workspace = "prod"
    config = {
        bucket = "myorg-terraform-state"
        key    = "networking/terraform.tfstate"
        region =  "eu-west-2"
    }
}

Based on the example commands you showed in your comment, I think your intend is to standardize on using the same workspace names across all of your configurations, and so you want Terraform to use the same workspace name as is selected for the current configuration. You can make Terraform do that automatically by setting the workspace argument to the symbol terraform.workspace, which returns the current workspace name:

data "terraform_remote_state" "networking" {
    backend   = "s3"
    workspace = terraform.workspace
    config = {
        bucket = "myorg-terraform-state"
        key    = "networking/terraform.tfstate"
        region =  "eu-west-2"
    }
}

The terraform_remote_state data source selects the “default” workspace by default because it’s expecting the use-cases for multiple workspaces described in When to use Multiple Workspaces, where the main workspace for each configuration is called default and other workspaces exist only temporarily for testing. You can use the terraform.workspace symbol as I showed above to use non-default workspaces consistently across all of your configurations, but I’d suggest reviewing the advice in that section to decide whether that is the best design for your situation or whether you’d rather use one of the alternatives discussed in the documentation.