Apply - Error updating organization citrusoft: resource not found

citrusoft · May 2, 2023, 2:03pm

My strategy is to use the organization(citrusoft), project(iam-kiosk) and workspace(infrastructure) and a list of account #s to create workspace per account.
Here is the root module.

provider "tfe" {
  hostname = var.tfe_hostname
  organization = "citrusoft"
}

data "tfe_organization" "citrusoft" {
  name  = "citrusoft"
}
...
# Module used to create the workspace, add workspace variables, and teams access.
module "workspaces" {
  source            = "./modules/workspaces"
  for_each          = local.workspaces
  name              = each.key
  organization      = data.tfe_organization.citrusoft.name
  queue_all_runs    = each.value.queue_all_runs
  auto_apply        = each.value.auto_apply
  working_directory = each.value.working_directory
  tf_vars           = each.value.tf_vars
  varset            = each.value.varset
  var_file          = each.value.var_file
  parallelism       = each.value.parallelism
  tag_names         = setunion(each.value.tags, [each.value.environment])
  env_vars          = each.value.env_vars
  policy_env        = each.value.policy_env
  terraform_version = each.value.terraform_version
  drift_detection   = each.value.drift_detection
  vcs_repo = {
    branch             = each.value.branch
    identifier         = "${each.value.github_org}/${each.value.github_repo}"
    ingress_submodules = try(each.value.ingress_submodules, false)
    oauth_token_id     = var.oauth_token_id
  }
}

I created the organization and workspace via the Browser console and I imported the organization into the workspace(infrastructure).

$ terraform import data.tfe_organization.citrusoft citrusoft
$ terraform state list
null_resource.list-files
tfe_organization.citrusoft

Given my token is generated from my user identity whom is a member of owners team,
And the above state
And the above root module
When I perform “terraform apply --auto-approve”
Then I get the following response…

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # tfe_organization.citrusoft will be updated in-place
  ~ resource "tfe_organization" "citrusoft" {
      ~ allow_force_delete_workspaces                           = true -> false
        id                                                      = "citrusoft"
        name                                                    = "citrusoft"
        # (8 unchanged attributes hidden)
    }

  # module.workspaces["123133550781"].tfe_workspace.this will be created
  + resource "tfe_workspace" "this" {
      + allow_destroy_plan            = true
      + auto_apply                    = false
      + execution_mode                = "remote"
      + file_triggers_enabled         = true
      + force_delete                  = false
      + global_remote_state           = (known after apply)
      + id                            = (known after apply)
      + name                          = "123133550781"
      + operations                    = (known after apply)
      + organization                  = "citrusoft"
      + project_id                    = (known after apply)
      + queue_all_runs                = true
      + remote_state_consumer_ids     = (known after apply)
      + resource_count                = (known after apply)
      + speculative_enabled           = true
      + structured_run_output_enabled = true
      + tag_names                     = [
          + "environment",
        ]
      + terraform_version             = (known after apply)
    }

  # module.workspaces["930856341568"].tfe_workspace.this will be created
  + resource "tfe_workspace" "this" {
      + allow_destroy_plan            = true
      + auto_apply                    = false
      + execution_mode                = "remote"
      + file_triggers_enabled         = true
      + force_delete                  = false
      + global_remote_state           = (known after apply)
      + id                            = (known after apply)
      + name                          = "930856341568"
      + operations                    = (known after apply)
      + organization                  = "citrusoft"
      + project_id                    = (known after apply)
      + queue_all_runs                = true
      + remote_state_consumer_ids     = (known after apply)
      + resource_count                = (known after apply)
      + speculative_enabled           = true
      + structured_run_output_enabled = true
      + tag_names                     = [
          + "environment",
        ]
      + terraform_version             = (known after apply)
    }

Plan: 2 to add, 1 to change, 0 to destroy.

tfe_organization.citrusoft: Modifying... [id=citrusoft]
╷
│ Error: Error updating organization citrusoft: resource not found
│ 
│   with tfe_organization.citrusoft,
│   on main.tf line 19, in resource "tfe_organization" "citrusoft":
│   19: resource "tfe_organization" "citrusoft" {
│ 
╵
Operation failed: failed running terraform apply (exit 1)

What am I doing wrong?

maxb · May 2, 2023, 10:26pm

You have edited this question since posting it, to change from using a tfe_organization resource, to data source instead.

However, with this edit, it no longer makes sense:

You don’t terraform import data sources
You haven’t edited the error message you are asking about, which still reflects the original configuration using resource. You won’t get the same error now.

maxb · May 2, 2023, 10:39pm

I tried out your original configuration on my own Terraform Cloud org. It worked without this error. I’d be inclined to suspect the TFC token didn’t have enough privileges … but you said

You could attempt the apply with extremely verbose debug logging turned on - the environment variable TF_LOG_PROVIDER=TRACE - perhaps more specifics about the failure might be revealed.

citrusoft · May 3, 2023, 2:55pm

So, I believe that you may be onto the root-cause.
I generated my “User Token” using my User whom has “owner” privies thus I would expect this token to have “god” privileges. So, I modified code to simply create an organization and workspace.

provider "tfe" {
  hostname = var.tfe_hostname
  organization = var.organization
}

resource "tfe_organization" "citrusoft" {
  name  = var.organization
  email = "thunt@citrusoft.org"
}
...
module "workspaces" {
  source            = "./modules/workspaces"
  for_each          = local.workspaces
  name              = each.key
  organization      = var.organization
  queue_all_runs    = each.value.queue_all_runs
  auto_apply        = each.value.auto_apply
  working_directory = each.value.working_directory
  tf_vars           = each.value.tf_vars
  varset            = each.value.varset
  var_file          = each.value.var_file
  parallelism       = each.value.parallelism
  tag_names         = [ "environment" ]
  policy_env        = each.value.policy_env
  terraform_version = each.value.terraform_version
  drift_detection   = each.value.drift_detection
}

When I perform “terraform apply --auto-approve”, then I get another set of errors…

Plan: 3 to add, 0 to change, 0 to destroy.

tfe_organization.citrusoft: Creating...
module.workspaces["930856341568"].tfe_workspace.this: Creating...
module.workspaces["123133550781"].tfe_workspace.this: Creating...
╷
│ Error: Error creating workspace 930856341568 for organization citrusoft3: resource not found
│ 
│   with module.workspaces["930856341568"].tfe_workspace.this,
│   on modules/workspaces/main.tf line 25, in resource "tfe_workspace" "this":
│   25: resource "tfe_workspace" "this" {
│ 
╵
╷
│ Error: Error creating the new organization citrusoft3: forbidden
│ 
│ Creation of organizations has been disabled for non-admins.
│ 
│   with tfe_organization.citrusoft,
│   on main.tf line 19, in resource "tfe_organization" "citrusoft":
│   19: resource "tfe_organization" "citrusoft" {
│ 
╵
╷
│ Error: Error creating workspace 123133550781 for organization citrusoft3: resource not found
│ 
│   with module.workspaces["123133550781"].tfe_workspace.this,
│   on modules/workspaces/main.tf line 25, in resource "tfe_workspace" "this":
│   25: resource "tfe_workspace" "this" {
│ 
╵
Operation failed: failed running terraform apply (exit 1)

The documentation is confusing. How do I solve this privileges issue?

citrusoft · May 3, 2023, 5:48pm

@maxb thanks for your suggestions.
I use “export TF_LOG=TRACE” to trigger the logger to render messages.

$ terraform apply -auto-approve
2023-05-03T13:12:26.001-0400 [INFO]  Terraform version: 1.4.6
2023-05-03T13:12:26.002-0400 [DEBUG] using github.com/hashicorp/go-tfe v1.21.0
2023-05-03T13:12:26.002-0400 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.2
2023-05-03T13:12:26.002-0400 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2023-05-03T13:12:26.002-0400 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.0
2023-05-03T13:12:26.002-0400 [DEBUG] using github.com/zclconf/go-cty v1.12.1
2023-05-03T13:12:26.002-0400 [INFO]  Go runtime version: go1.19.6
2023-05-03T13:12:26.002-0400 [INFO]  CLI args: []string{"terraform", "apply", "-auto-approve"}
2023-05-03T13:12:26.002-0400 [TRACE] Stdout is a terminal of width 129
2023-05-03T13:12:26.002-0400 [TRACE] Stderr is a terminal of width 129
2023-05-03T13:12:26.002-0400 [TRACE] Stdin is a terminal
2023-05-03T13:12:26.002-0400 [DEBUG] Attempting to open CLI config file: /Users/thunt/.terraformrc
2023-05-03T13:12:26.002-0400 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-05-03T13:12:26.002-0400 [INFO]  Loading CLI configuration from /Users/thunt/.terraform.d/credentials.tfrc.json
2023-05-03T13:12:26.003-0400 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-05-03T13:12:26.003-0400 [DEBUG] ignoring non-existing provider search directory /Users/thunt/.terraform.d/plugins
2023-05-03T13:12:26.003-0400 [DEBUG] ignoring non-existing provider search directory /Users/thunt/Library/Application Support/io.terraform/plugins
2023-05-03T13:12:26.003-0400 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-05-03T13:12:26.003-0400 [INFO]  CLI command args: []string{"apply", "-auto-approve"}
2023-05-03T13:12:26.007-0400 [TRACE] Meta.Backend: built configuration for "cloud" backend with hash value 4043148377
2023-05-03T13:12:26.007-0400 [TRACE] Preserving existing state lineage "d7defe5a-7c88-32b0-43b8-9b1c83d19657"
2023-05-03T13:12:26.007-0400 [TRACE] Preserving existing state lineage "d7defe5a-7c88-32b0-43b8-9b1c83d19657"
2023-05-03T13:12:26.008-0400 [TRACE] Meta.Backend: working directory was previously initialized for "cloud" backend
2023-05-03T13:12:26.008-0400 [TRACE] Meta.Backend: using already-initialized, unchanged "cloud" backend configuration
2023-05-03T13:12:26.008-0400 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2023-05-03T13:12:26.008-0400 [TRACE] HTTP client GET request to https://app.terraform.io/.well-known/terraform.json
2023-05-03T13:12:26.340-0400 [DEBUG] Service discovery for app.terraform.io aliased as localterraform.com
2023-05-03T13:12:26.782-0400 [TRACE] Meta.Backend: instantiated backend of type *cloud.Cloud
2023-05-03T13:12:26.783-0400 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-05-03T13:12:26.785-0400 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/null v3.2.1 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/null/3.2.1/darwin_amd64
2023-05-03T13:12:26.786-0400 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/tfe v0.42.0 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/tfe/0.42.0/darwin_amd64
2023-05-03T13:12:26.786-0400 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/null/3.2.1/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/null 3.2.1
2023-05-03T13:12:26.786-0400 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/tfe/0.42.0/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/tfe 0.42.0
2023-05-03T13:12:26.841-0400 [DEBUG] checking for provisioner in "."
2023-05-03T13:12:26.842-0400 [DEBUG] checking for provisioner in "/Users/thunt/bin"
2023-05-03T13:12:26.842-0400 [TRACE] Meta.Backend: backend *cloud.Cloud supports operations
2023-05-03T13:12:26.976-0400 [INFO]  cloud: starting Apply operation
Running apply in Terraform Cloud. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://app.terraform.io/app/citrusoft/infrastructure/runs/run-YZmU86GzVhoKSK3X

Waiting for the plan to start...

Terraform v1.4.6
on linux_amd64
Initializing plugins and modules...
data.tfe_organization.citrusoft: Refreshing...
data.tfe_organization.citrusoft: Refresh complete after 0s [id=org-CKkoi5HWu7uqB8Pc]
╷
│ Warning: Value for undeclared variable
│ 
│ The root module does not declare a variable named "AWS_ACCESS_KEY_ID" but a
│ value was found in file
│ "/home/tfc-agent/.tfc-agent/component/terraform/runs/run-YZmU86GzVhoKSK3X/terraform.tfvars".
│ If you meant to use this value, add a "variable" block to the
│ configuration.
...
**Plan:** 2 to add, 0 to change, 0 to destroy.
**module.workspaces["930856341568"].tfe_workspace.this: Creating...**
**module.workspaces["123133550781"].tfe_workspace.this: Creating...**
╷
│ **Error:** **Error creating workspace 123133550781 for organization citrusoft: resource not found**
│
│ with module.workspaces["123133550781"].tfe_workspace.this,
│ on modules/workspaces/main.tf line 25, in resource "tfe_workspace" "this":
│ 25: resource "tfe_workspace" "this" {
│
╵
╷
│ **Error:** **Error creating workspace 930856341568 for organization citrusoft: resource not found**
│
│ with module.workspaces["930856341568"].tfe_workspace.this,
│ on modules/workspaces/main.tf line 25, in resource "tfe_workspace" "this":
│ 25: resource "tfe_workspace" "this" {
╵

Operation failed: failed running terraform apply (exit 1)

I am beginning to think that THE ORGANIZATION NOTHING TO DO WITH THE ERROR MESSAGES.

Why is it looking for a variable, AWS_ACCESS_KEY_ID, ? Nothing in here has anything to do with AWS provider.

Could there be something wrong with my submodule specification?

here is my terraform init…

$ terraform init
2023-05-03T13:40:25.062-0400 [INFO]  Terraform version: 1.4.6
2023-05-03T13:40:25.063-0400 [DEBUG] using github.com/hashicorp/go-tfe v1.21.0
2023-05-03T13:40:25.063-0400 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.2
2023-05-03T13:40:25.063-0400 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2023-05-03T13:40:25.063-0400 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.0
2023-05-03T13:40:25.063-0400 [DEBUG] using github.com/zclconf/go-cty v1.12.1
2023-05-03T13:40:25.063-0400 [INFO]  Go runtime version: go1.19.6
2023-05-03T13:40:25.063-0400 [INFO]  CLI args: []string{"terraform", "init"}
2023-05-03T13:40:25.063-0400 [TRACE] Stdout is a terminal of width 129
2023-05-03T13:40:25.063-0400 [TRACE] Stderr is a terminal of width 129
2023-05-03T13:40:25.063-0400 [TRACE] Stdin is a terminal
2023-05-03T13:40:25.063-0400 [DEBUG] Attempting to open CLI config file: /Users/thunt/.terraformrc
2023-05-03T13:40:25.063-0400 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-05-03T13:40:25.063-0400 [INFO]  Loading CLI configuration from /Users/thunt/.terraform.d/credentials.tfrc.json
2023-05-03T13:40:25.063-0400 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-05-03T13:40:25.063-0400 [DEBUG] ignoring non-existing provider search directory /Users/thunt/.terraform.d/plugins
2023-05-03T13:40:25.064-0400 [DEBUG] ignoring non-existing provider search directory /Users/thunt/Library/Application Support/io.terraform/plugins
2023-05-03T13:40:25.064-0400 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-05-03T13:40:25.064-0400 [INFO]  CLI command args: []string{"init"}

Initializing Terraform Cloud...
2023-05-03T13:40:25.067-0400 [TRACE] Meta.Backend: built configuration for "cloud" backend with hash value 4043148377
2023-05-03T13:40:25.067-0400 [TRACE] Preserving existing state lineage "d7defe5a-7c88-32b0-43b8-9b1c83d19657"
2023-05-03T13:40:25.067-0400 [TRACE] Preserving existing state lineage "d7defe5a-7c88-32b0-43b8-9b1c83d19657"
2023-05-03T13:40:25.068-0400 [TRACE] Meta.Backend: working directory was previously initialized for "cloud" backend
2023-05-03T13:40:25.068-0400 [TRACE] Meta.Backend: using already-initialized, unchanged "cloud" backend configuration
2023-05-03T13:40:25.068-0400 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2023-05-03T13:40:25.068-0400 [TRACE] HTTP client GET request to https://app.terraform.io/.well-known/terraform.json
2023-05-03T13:40:25.442-0400 [DEBUG] Service discovery for app.terraform.io aliased as localterraform.com
2023-05-03T13:40:26.080-0400 [TRACE] Meta.selectWorkspace: the currently selected workspace is present in the configured backend (infrastructure)
2023-05-03T13:40:26.080-0400 [TRACE] Meta.Backend: instantiated backend of type *cloud.Cloud
2023-05-03T13:40:26.082-0400 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-05-03T13:40:26.083-0400 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/null v3.2.1 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/null/3.2.1/darwin_amd64
2023-05-03T13:40:26.083-0400 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/tfe v0.42.0 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/tfe/0.42.0/darwin_amd64
2023-05-03T13:40:26.083-0400 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/null/3.2.1/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/null 3.2.1
2023-05-03T13:40:26.083-0400 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/tfe/0.42.0/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/tfe 0.42.0
2023-05-03T13:40:26.143-0400 [DEBUG] checking for provisioner in "."
2023-05-03T13:40:26.143-0400 [DEBUG] checking for provisioner in "/Users/thunt/bin"
2023-05-03T13:40:26.143-0400 [TRACE] Meta.Backend: backend *cloud.Cloud supports operations
Initializing modules...
2023-05-03T13:40:26.950-0400 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
2023-05-03T13:40:26.952-0400 [DEBUG] Module installer: begin workspaces
2023-05-03T13:40:26.956-0400 [TRACE] ModuleInstaller: Module installer: workspaces <nil> already installed in modules/workspaces
2023-05-03T13:40:26.956-0400 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

Initializing provider plugins...
- Reusing previous version of hashicorp/tfe from the dependency lock file
2023-05-03T13:40:26.961-0400 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2023-05-03T13:40:26.961-0400 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2023-05-03T13:40:27.089-0400 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/tfe/versions
2023-05-03T13:40:27.089-0400 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/tfe/versions
2023-05-03T13:40:27.191-0400 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-05-03T13:40:27.192-0400 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/null v3.2.1 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/null/3.2.1/darwin_amd64
2023-05-03T13:40:27.193-0400 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/tfe v0.42.0 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/tfe/0.42.0/darwin_amd64
2023-05-03T13:40:27.193-0400 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/null/3.2.1/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/null 3.2.1
2023-05-03T13:40:27.193-0400 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/tfe/0.42.0/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/tfe 0.42.0
- Using previously-installed hashicorp/tfe v0.42.0

Terraform Cloud has been successfully initialized!

You may now begin working with Terraform Cloud. Try running "terraform plan" to
see any changes that are required for your infrastructure.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

Here is my submodule…

# Create the workspace, line number 24
resource "tfe_workspace" "this" {
  name                  = var.name
  organization          = var.organization
  tag_names             = [for tag in var.tag_names : lower(tag)]
}

Hmm… what does this error message mean?

Plan: 2 to add, 0 to change, 0 to destroy.

module.workspaces["123133550781"].tfe_workspace.this: Creating...
module.workspaces["930856341568"].tfe_workspace.this: Creating...
╷
│ Error: Error creating workspace 123133550781 for organization citrusoft: resource not found
│ 
│   with module.workspaces["123133550781"].tfe_workspace.this,
│   on modules/workspaces/main.tf line 25, in resource "tfe_workspace" "this":
│   25: resource "tfe_workspace" "this" {
│ 
╵
╷
│ Error: Error creating workspace 930856341568 for organization citrusoft: resource not found
│ 
│   with module.workspaces["930856341568"].tfe_workspace.this,
│   on modules/workspaces/main.tf line 25, in resource "tfe_workspace" "this":
│   25: resource "tfe_workspace" "this" {
│ 
╵
Operation failed: failed running terraform apply (exit 1)

qwerty1979bg · May 4, 2023, 10:55pm

Hi @citrusoft ,

The AWS_ACCESS_KEY_ID is a variable that is probably set in your workspace inside Terraform Cloud. Perhaps as part of a variable set that is applied to many / all workspaces ?

While there is no point to have it in this workspace, you can ignore this warning, as it will also not interfere with your TFE provider resources.

I would suggest to start with simple code - a single tfe_workspace resource, provide the TFE provider with an authentication token (for example, via an ENV variable named TFE_TOKEN in the respective Terraform Cloud workspace) and once this works, start adding up to your Terraform code.

Regards,
Filip

citrusoft · May 5, 2023, 5:15pm

Solved: Add workspace environment variable TFE_TOKEN.

A newbie mistake but there is some important feedback here…
The terraform error message says “resource not found” whereas I’ll bet that the API returns HTTP status code 401 or 404. Thus, the exception / error handling needs improvement.

How do I go about providing this feedback to the developers?

Also, how do I mark this reply to the POST as the correct resolution?

maxb · May 5, 2023, 5:31pm

A GitHub issue on terraform-provider-tfe is probably the best way.