I’m getting the following error when running terraform plan (or apply) from any of my machines for a specific workspace:
Waiting for the plan to start...
Terraform v1.3.9
on linux_amd64
Initializing plugins and modules...
Initializing modules...
Initializing Terraform Cloud...
Error loading state: Error updating workspace My-Workspace: resource not found
Operation failed: failed running terraform init (exit 1)
I’m using Terraform Cloud workspaces with remote execution and Github VCS connection. I just recently made a slew of workspace setting changes, including replacing the Github App integration with the Oauth Integration, and switching from ‘prefix’ change detection to pattern. Now, nothing works from the CLI on any of my machines for this workspace. Github integration works perfectly still, and my other workspaces work also. I tried reverting all of the settings changes I made, but no luck.
Something seems corrupted with this workspace. Is this possible to fix, or do I need to delete/recreate the workspace?
I just created a new Workspace with no state with the same Tags (but in a different Terraform Cloud Project, the default project) and its experiencing the same bug…
I also just generated all new CLI access tokens for my account. I don’t think thats the issue though, because my CLI run is visible in the Dashboard just fine, so I’m authenticated to TF Cloud. But its like the TF Cloud managed runners are failing to authorize, just from CLI triggered runs?
I’m not familiar with this error but I looked in the Terraform codebase for it and I believe it’s originating here:
It seems that the Terraform Cloud integration has decided that this workspace’s tags need updating, but is failing to successfully perform that update.
It seems that the rule for whether workspace tags need updating is when your cloud block specifies a tag that isn’t present on your selected workspace. Perhaps then you might be able to avoid the error by making sure your workspace tags match between the cloud block and the real remote workspace.
With that said, I’m not sure why adding a tag like this would return “resource not found”, so it does seem like something strange is going on here.
I expect this is something that won’t be fully resolvable here in the community forum because it seems to involve some specifics of how your configuration is written and how your workspace is set up, and so I’d recommend contacting HashiCorp Support for help with this problem. Unlike participants in this forum, HashiCorp Support can (with your permission) look at exactly how your workspace is configured and so might notice something that I wouldn’t think to ask about.
@apparentlymart Do you have any guidance as to what could cause the tags to be updated? I tried to make sense of that function but couldn’t, and terraform support is being quite unhelpful.
My workspace tags are quite simple. I have 1 tag applied to two workspaces.
I’m afraid without being able to directly inspect your workspaces I don’t have a theory as to why it would behave this way: Terraform CLI is using the tags to filter which workspaces are included so by definition any workspace you can select should have those tags.
Therefore I think there is a bug somewhere, but I’m not sure where. The fact that when it tries to add the tags it gets an error is also very strange: adding tags to the workspace should work and should be idempotent if they were already set.
I was hoping that the support team would be able to use their access to the running system (which I don’t have) to make sure all of the objects involved are in a consistent state; without being able to see how everything is set up in the live system it’s hard to guess what might be wrong.
If you did hear anything new from the support team that you would be comfortable sharing here then that might give me or someone else a new clue for something to ask about.
I am experiencing the same error message…
I turned on logging at the debug and trace levels but not helpful log messages.
export TF_LOG_PROVIDER=DEBUG
export TF_LOG=DEBUG
$ terraform apply -auto-approve -compact-warnings
2023-05-04T13:55:01.110-0700 [INFO] Terraform version: 1.4.6
2023-05-04T13:55:01.111-0700 [DEBUG] using github.com/hashicorp/go-tfe v1.21.0
2023-05-04T13:55:01.111-0700 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.2
2023-05-04T13:55:01.111-0700 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2023-05-04T13:55:01.111-0700 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.0
2023-05-04T13:55:01.111-0700 [DEBUG] using github.com/zclconf/go-cty v1.12.1
2023-05-04T13:55:01.111-0700 [INFO] Go runtime version: go1.19.6
2023-05-04T13:55:01.111-0700 [INFO] CLI args: []string{"/usr/local/Cellar/tfenv/3.0.0/versions/1.4.6/terraform", "apply", "-auto-approve", "-compact-warnings"}
2023-05-04T13:55:01.111-0700 [DEBUG] Attempting to open CLI config file: /Users/TAHV/.terraformrc
2023-05-04T13:55:01.111-0700 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-05-04T13:55:01.111-0700 [INFO] Loading CLI configuration from /Users/TAHV/.terraform.d/credentials.tfrc.json
2023-05-04T13:55:01.112-0700 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-05-04T13:55:01.112-0700 [DEBUG] ignoring non-existing provider search directory /Users/TAHV/.terraform.d/plugins
2023-05-04T13:55:01.112-0700 [DEBUG] ignoring non-existing provider search directory /Users/TAHV/Library/Application Support/io.terraform/plugins
2023-05-04T13:55:01.112-0700 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-05-04T13:55:01.112-0700 [INFO] CLI command args: []string{"apply", "-auto-approve", "-compact-warnings"}
2023-05-04T13:55:01.130-0700 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2023-05-04T13:55:02.987-0700 [DEBUG] Service discovery for app.terraform.io aliased as localterraform.com
2023-05-04T13:55:04.159-0700 [DEBUG] checking for provisioner in "."
2023-05-04T13:55:04.159-0700 [DEBUG] checking for provisioner in "/usr/local/Cellar/tfenv/3.0.0/versions/1.4.6"
2023-05-04T13:55:04.496-0700 [INFO] cloud: starting Apply operation
Running apply in Terraform Cloud. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.
Preparing the remote apply...
To view this run in a browser, visit:
https://app.terraform.io/app/citrusoft/infrastructure/runs/run-2LAeiiVLeV4T4ABX
Waiting for the plan to start...
Terraform v1.4.6
on linux_amd64
Initializing plugins and modules...
data.tfe_organization.citrusoft: Refreshing...
data.tfe_organization.citrusoft: Refresh complete after 0s [id=org-CKkoi5HWu7uqB8Pc]
╷
│ Warning: Value for undeclared variable
│
│ The root module does not declare a variable named "AWS_ACCESS_KEY_ID" but a
│ value was found in file
│ "/home/tfc-agent/.tfc-agent/component/terraform/runs/run-2LAeiiVLeV4T4ABX/terraform.tfvars".
│ If you meant to use this value, add a "variable" block to the
│ configuration.
│
│ To silence these warnings, use TF_VAR_... environment variables to provide
│ certain "global" settings to all configurations in your organization. To
│ reduce the verbosity of these warnings, use the -compact-warnings option.
╵
╷
│ Warning: Value for undeclared variable
│
│ The root module does not declare a variable named "AWS_SECRET_ACCESS_KEY"
│ but a value was found in file
│ "/home/tfc-agent/.tfc-agent/component/terraform/runs/run-2LAeiiVLeV4T4ABX/terraform.tfvars".
│ If you meant to use this value, add a "variable" block to the
│ configuration.
│
│ To silence these warnings, use TF_VAR_... environment variables to provide
│ certain "global" settings to all configurations in your organization. To
│ reduce the verbosity of these warnings, use the -compact-warnings option.
╵
2023-05-04T13:55:19.445-0700 [DEBUG] performing request: method=GET url=https://app.terraform.io/api/v2/plans/plan-hXCpS7ikVt4Xkkqh/json-output-redacted
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# tfe_workspace.test will be created
+ resource "tfe_workspace" "test" {
+ allow_destroy_plan = true
+ auto_apply = false
+ execution_mode = "remote"
+ file_triggers_enabled = true
+ force_delete = false
+ global_remote_state = (known after apply)
+ html_url = (known after apply)
+ id = (known after apply)
+ name = "my-workspace-name"
+ operations = (known after apply)
+ organization = "citrusoft"
+ project_id = (known after apply)
+ queue_all_runs = true
+ remote_state_consumer_ids = (known after apply)
+ resource_count = (known after apply)
+ speculative_enabled = true
+ structured_run_output_enabled = true
+ tag_names = [
+ "app",
+ "test",
]
+ terraform_version = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
tfe_workspace.test: Creating...
╷
│ Error: Error creating workspace my-workspace-name for organization citrusoft: resource not found
│
│ with tfe_workspace.test,
│ on main.tf line 29, in resource "tfe_workspace" "test":
│ 29: resource "tfe_workspace" "test" {
│
╵
Operation failed: failed running terraform apply (exit 1)
╵
Operation failed: failed running terraform apply (exit 1)
However, in my case, I am trying to create the workspace.
I have tried new organization, new parent workspace, new computer, new software install but they all reply with the same error message.
I created my User API token where I am a member of the owners team.
I would expect that I am privileged enough.
The code is super SIMPLE…
terraform {
required_version = ">= 1.4.5"
required_providers {
tfe = {
source = "hashicorp/tfe"
version = "~> 0.44.0"
}
}
cloud {
organization = "citrusoft"
workspaces {
name = "infrastructure"
}
}
}
provider "tfe" {
hostname = var.tfe_hostname
organization = var.organization
}
data "tfe_organization" "citrusoft" {
name = var.organization
}
resource "tfe_workspace" "test" {
name = "my-workspace-name"
organization = data.tfe_organization.citrusoft.name
tag_names = ["test", "app"]
}
variable "tfe_hostname" {
description = "The Terraform Enterprise hostname to connect to. Defaults to `app.terraform.io`. Can be overridden by setting the `TFE_HOSTNAME` environment variable."
default = "app.terraform.io"
type = string
}
variable "organization" {
description = "Name of the organization."
type = string
default = "citrusoft"
}
After 25 days, what did you learn? how did you fix or workaround it?
While the error messages might be similar, the issues seem different.
You are starting a Terraform Cloud run via Terraform CLI.
As the run is happening inside Terraform Cloud’s ephemeral worker VMs, it does not have access to the token you have provided to your Terraform CLI.
I think you would need to provide an authentication token to the TFE provider (for example, via an ENV variable named TFE_TOKEN in the respective Terraform Cloud workspace).
Thanks for following up. I actually had a very long and thorough call with support, and we were able to isolate/reproduce my issue. It’s a case-sensitivity of the TF Cloud workspace API!
Doesn’t work :
tags = ["My-App"]
Does work :
tags = ["my-app"]
Support told me that since its a reproducible regression, they would open a support ticket internally. I reached out to them for the ticket ID, but could you please check if such a ticket exists?
Here is my solution for another situation returning the same error message…
Add workspace environment variable TFE_TOKEN so the workspace will have credentials to do the work.
A newbie mistake but there is some important feedback here…
The terraform error message says “resource not found” whereas I’ll bet that the API returns HTTP status code 401 or 404. Thus, the exception / error handling needs improvement.
Much of the Terraform Cloud API follows the typical convention of returning “not found” both for a missing object and for an object you don’t have access to, to avoid exposing metadata about what exists and what does not.
Unfortunately that does mean poor feedback in situations like yours: the client doesn’t have any context about what happened, so it can only return a generic error message.
This particular case is trickier still because Terraform Cloud’s execution environment automatically provides a TFE_TOKEN environment variable whose token has limited access, so in your situation Terraform CLI was accessing the Terraform Cloud API using a valid token with insufficient access, rather than attempting an anonymous request.
Thanks for reporting the problem to the support team! I can see the ticket you mentioned and it has found its way to the Terraform Cloud team which is responsible for the integration between Terraform CLI and Terraform Cloud, but at the time I’m writing this it’s waiting in their queue for review.
Thanks for sharing the solution - I’ve been scratching my head on this one since yesterday… I too was focused on finding what was missing - because nothing appeared to be.
:
: